  Federated learning (FL) is an emerging machine learning paradigm in which a
central server coordinates multiple participants (clients) collaboratively to
train on decentralized data. In practice, FL often faces statistical, system,
and model heterogeneities, which inspires the field of Model-Heterogeneous
Personalized Federated Learning (MHPFL). With the increased interest in
adopting large language models (LLMs) in FL, the existing MHPFL methods cannot
achieve acceptable computational and communication costs, while maintaining
satisfactory model performance. To bridge this gap, we propose a novel and
efficient model-heterogeneous personalized Federated learning framework based
on LoRA tuning (pFedLoRA). Inspired by the popular LoRA method for fine-tuning
pre-trained LLMs with a low-rank model (a.k.a., an adapter), we design a
homogeneous small adapter to facilitate federated client's heterogeneous local
model training with our proposed iterative training for global-local knowledge
exchange. The homogeneous small local adapters are aggregated on the FL server
to generate a global adapter. We theoretically prove the convergence of
pFedLoRA. Extensive experiments on two benchmark datasets demonstrate that
pFedLoRA outperforms six state-of-the-art baselines, beating the best method by
1.35% in test accuracy, 11.81 times computation overhead reduction and 7.41
times communication cost saving.
