  Federated learning (FL) is a framework for users to jointly train a machine
learning model. FL is promoted as a privacy-enhancing technology (PET) that
provides data minimization: data never "leaves" personal devices and users
share only model updates with a server (e.g., a company) coordinating the
distributed training. While prior work showed that in vanilla FL a malicious
server can extract users' private data from the model updates, in this work we
take it further and demonstrate that a malicious server can reconstruct user
data even in hardened versions of the protocol. More precisely, we propose an
attack against FL protected with distributed differential privacy (DDP) and
secure aggregation (SA). Our attack method is based on the introduction of
sybil devices that deviate from the protocol to expose individual users' data
for reconstruction by the server. The underlying root cause for the
vulnerability to our attack is a power imbalance: the server orchestrates the
whole protocol and users are given little guarantees about the selection of
other users participating in the protocol. Moving forward, we discuss
requirements for privacy guarantees in FL. We conclude that users should only
participate in the protocol when they trust the server or they apply local
primitives such as local DP, shifting power away from the server. Yet, the
latter approaches come at significant overhead in terms of performance
degradation of the trained model, making them less likely to be deployed in
practice.
