  Increasing concerns and regulations about data privacy and sparsity
necessitate the study of privacy-preserving, decentralized learning methods for
natural language processing (NLP) tasks. Federated learning (FL) provides
promising approaches for a large number of clients (e.g., personal devices or
organizations) to collaboratively learn a shared global model to benefit all
clients while allowing users to keep their data locally. Despite interest in
studying FL methods for NLP tasks, a systematic comparison and analysis is
lacking in the literature. Herein, we present the FedNLP, a benchmarking
framework for evaluating federated learning methods on four different task
formulations: text classification, sequence tagging, question answering, and
seq2seq. We propose a universal interface between Transformer-based language
models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under
various non-IID partitioning strategies. Our extensive experiments with FedNLP
provide empirical comparisons between FL methods and helps us better understand
the inherent challenges of this direction. The comprehensive analysis points to
intriguing and exciting future research aimed at developing FL methods for NLP
tasks.
