  Federated Learning (FL) systems are gaining popularity as a solution to
training Machine Learning (ML) models from large-scale user data collected on
personal devices (e.g., smartphones) without their raw data leaving the device.
At the core of FL is a network of anonymous user devices sharing training
information (model parameter updates) computed locally on personal data.
However, the type and degree to which user-specific information is encoded in
the model updates is poorly understood. In this paper, we identify model
updates encode subtle variations in which users capture and generate data. The
variations provide a strong statistical signal, allowing an adversary to
effectively deanonymize participating devices using a limited set of auxiliary
data. We analyze resulting deanonymization attacks on diverse tasks on
real-world (anonymized) user-generated data across a range of closed- and
open-world scenarios. We study various strategies to mitigate the risks of
deanonymization. As random perturbation methods do not offer convincing
operating points, we propose data-augmentation strategies which introduces
adversarial biases in device data and thereby, offer substantial protection
against deanonymization threats with little effect on utility.
