  Federated Learning (FL) exhibits privacy vulnerabilities under gradient
inversion attacks (GIAs), which can extract private information from individual
gradients. To enhance privacy, FL incorporates Secure Aggregation (SA) to
prevent the server from obtaining individual gradients, thus effectively
resisting GIAs. In this paper, we propose a stealthy label inference attack to
bypass SA and recover individual clients' private labels. Specifically, we
conduct a theoretical analysis of label inference from the aggregated gradients
that are exclusively obtained after implementing SA. The analysis results
reveal that the inputs (embeddings) and outputs (logits) of the final fully
connected layer (FCL) contribute to gradient disaggregation and label
restoration. To preset the embeddings and logits of FCL, we craft a fishing
model by solely modifying the parameters of a single batch normalization (BN)
layer in the original model. Distributing client-specific fishing models, the
server can derive the individual gradients regarding the bias of FCL by
resolving a linear system with expected embeddings and the aggregated gradients
as coefficients. Then the labels of each client can be precisely computed based
on preset logits and gradients of FCL's bias. Extensive experiments show that
our attack achieves large-scale label recovery with 100\% accuracy on various
datasets and model architectures.
