  Federated Learning (FL) is a setting for training machine learning models in
distributed environments where the clients do not share their raw data but
instead send model updates to a server. However, model updates can be subject
to attacks and leak private information. Differential Privacy (DP) is a leading
mitigation strategy which involves adding noise to clipped model updates,
trading off performance for strong theoretical privacy guarantees. Previous
work has shown that the threat model of DP is conservative and that the
obtained guarantees may be vacuous or may overestimate information leakage in
practice. In this paper, we aim to achieve a tighter measurement of the model
exposure by considering a realistic threat model. We propose a novel method,
CANIFE, that uses canaries - carefully crafted samples by a strong adversary to
evaluate the empirical privacy of a training round. We apply this attack to
vision models trained on CIFAR-10 and CelebA and to language models trained on
Sent140 and Shakespeare. In particular, in realistic FL scenarios, we
demonstrate that the empirical per-round epsilon obtained with CANIFE is 4-5x
lower than the theoretical bound.
