  Federated Learning (FL) is an emerging paradigm that enables distributed
users to collaboratively and iteratively train machine learning models without
sharing their private data. Motivated by the effectiveness and robustness of
self-attention-based architectures, researchers are turning to using
pre-trained Transformers (i.e., foundation models) instead of traditional
convolutional neural networks in FL to leverage their excellent transfer
learning capabilities. Despite recent progress, how pre-trained Transformer
models play a role in FL remains obscure, that is, how to efficiently fine-tune
these pre-trained models in FL and how FL users could benefit from this new
paradigm. In this paper, we explore this issue and demonstrate that the
fine-tuned Transformers achieve extraordinary performance on FL, and that the
lightweight fine-tuning method facilitates a fast convergence rate and low
communication costs. Concretely, we conduct a rigorous empirical study of three
tuning methods (i.e., modifying the input, adding extra modules, and adjusting
the backbone) using two types of pre-trained models (i.e., vision-language
models and vision models) for FL. Our experiments show that 1) Fine-tuning the
bias term of the backbone performs best when relying on a strong pre-trained
model; 2) The vision-language model (e.g., CLIP) outperforms the pure vision
model (e.g., ViT) and is more robust to the few-shot settings; 3) Compared to
pure local training, FL with pre-trained models has a higher accuracy because
it alleviates the problem of over-fitting. We will release our code and
encourage further exploration of pre-trained Transformers and FL.
