  Transformer-based pre-trained models have revolutionized NLP for superior
performance and generality. Fine-tuning pre-trained models for downstream tasks
often requires private data, for which federated learning is the de-facto
approach (i.e., FedNLP). However, our measurements show that FedNLP is
prohibitively slow due to the large model sizes and the resultant high
network/computation cost. Towards practical FedNLP, we identify as the key
building blocks adapters, small bottleneck modules inserted at a variety of
model layers. A key challenge is to properly configure the depth and width of
adapters, to which the training speed and efficiency is highly sensitive. No
silver-bullet configuration exists: the optimal choice varies across downstream
NLP tasks, desired model accuracy, and mobile resources. To automate adapter
configuration, we propose FedAdapter, a framework that enhances the existing
FedNLP with two novel designs. First, FedAdapter progressively upgrades the
adapter configuration throughout a training session; the principle is to
quickly learn shallow knowledge by only training fewer and smaller adapters at
the model's top layers, and incrementally learn deep knowledge by incorporating
deeper and larger adapters. Second, FedAdapter continuously profiles future
adapter configurations by allocating participant devices to trial groups.
Extensive experiments show that FedAdapter can reduce FedNLP's model
convergence delay to no more than several hours, which is up to 155.5$\times$
faster compared to vanilla FedNLP and 48$\times$ faster compared to strong
baselines.
