  Federated learning (FL) is a popular privacy-preserving paradigm that enables
distributed clients to collaboratively train models with a central server while
keeping raw data locally. In practice, distinct model architectures, varying
data distributions, and limited resources across local clients inevitably cause
model performance degradation and a slowdown in convergence speed. However,
existing FL methods can only solve some of the above heterogeneous challenges
and have obvious performance limitations. Notably, a unified framework has not
yet been explored to overcome these challenges. Accordingly, we propose FedHPL,
a parameter-efficient unified $\textbf{Fed}$erated learning framework for
$\textbf{H}$eterogeneous settings based on $\textbf{P}$rompt tuning and
$\textbf{L}$ogit distillation. Specifically, we employ a local prompt tuning
scheme that leverages a few learnable visual prompts to efficiently fine-tune
the frozen pre-trained foundation model for downstream tasks, thereby
accelerating training and improving model performance under limited local
resources and data heterogeneity. Moreover, we design a global logit
distillation scheme to handle the model heterogeneity and guide the local
training. In detail, we leverage logits to implicitly capture local knowledge
and design a weighted knowledge aggregation mechanism to generate global
client-specific logits. We provide a theoretical guarantee on the
generalization error bound for FedHPL. The experiments on various benchmark
datasets under diverse settings of models and data demonstrate that our
framework outperforms state-of-the-art FL approaches, with less computation
overhead and training rounds.
