  Deduplication is a vital preprocessing step that enhances machine learning
model performance and saves training time and energy. However, enhancing
federated learning through deduplication poses challenges, especially regarding
scalability and potential privacy violations if deduplication involves sharing
all clients' data. In this paper, we address the problem of deduplication in a
federated setup by introducing a pioneering protocol, Efficient
Privacy-Preserving Multi-Party Deduplication (EP-MPD). It efficiently removes
duplicates from multiple clients' datasets without compromising data privacy.
EP-MPD is constructed in a modular fashion, utilizing two novel variants of the
Private Set Intersection protocol. Our extensive experiments demonstrate the
significant benefits of deduplication in federated learning of large language
models. For instance, we observe up to 19.61% improvement in perplexity and up
to 27.95% reduction in running time. EP-MPD effectively balances privacy and
performance in federated learning, making it a valuable solution for
large-scale applications.
