  Recent works have shown that generative sequence models (e.g., language
models) have a tendency to memorize rare or unique sequences in the training
data. Since useful models are often trained on sensitive data, to ensure the
privacy of the training data it is critical to identify and mitigate such
unintended memorization. Federated Learning (FL) has emerged as a novel
framework for large-scale distributed learning tasks. However, it differs in
many aspects from the well-studied central learning setting where all the data
is stored at the central server. In this paper, we initiate a formal study to
understand the effect of different components of canonical FL on unintended
memorization in trained models, comparing with the central learning setting.
Our results show that several differing components of FL play an important role
in reducing unintended memorization. Specifically, we observe that the
clustering of data according to users---which happens by design in FL---has a
significant effect in reducing such memorization, and using the method of
Federated Averaging for training causes a further reduction. We also show that
training with a strong user-level differential privacy guarantee results in
models that exhibit the least amount of unintended memorization.
