  With the increasing availability of Foundation Models, federated tuning has
garnered attention in the field of federated learning, utilizing data and
computation resources from multiple clients to collaboratively fine-tune
foundation models. However, in real-world federated scenarios, there often
exist a multitude of heterogeneous clients with varying computation and
communication resources, rendering them incapable of supporting the entire
model fine-tuning process. In response to this challenge, we propose a novel
federated tuning algorithm, FedRA. The implementation of FedRA is
straightforward and can be seamlessly integrated into any transformer-based
model without the need for further modification to the original model.
Specifically, in each communication round, FedRA randomly generates an
allocation matrix. For resource-constrained clients, it reorganizes a small
number of layers from the original model based on the allocation matrix and
fine-tunes using adapters. Subsequently, the server aggregates the updated
adapter parameters from the clients according to the current allocation matrix
into the corresponding layers of the original model. It is worth noting that
FedRA also supports scenarios where none of the clients can support the entire
global model, which is an impressive advantage. We conduct experiments on two
large-scale image datasets, DomainNet and NICO++, under various non-iid
settings. The results demonstrate that FedRA outperforms the compared methods
significantly. The source code is available at
\url{https://github.com/leondada/FedRA}.
