  Federated learning (FL) offers privacy-preserving decentralized machine
learning, optimizing models at edge clients without sharing private data.
Simultaneously, foundation models (FMs) have gained traction in the artificial
intelligence (AI) community due to their exceptional performance across various
tasks. However, integrating FMs into FL presents challenges, primarily due to
their substantial size and intensive resource requirements. This is especially
true when considering the resource heterogeneity in edge FL systems. We present
an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to
address these challenges. RaFFM introduces specialized model compression
algorithms tailored for FL scenarios, such as salient parameter prioritization
and high-performance subnetwork extraction. These algorithms enable dynamic
scaling of given transformer-based FMs to fit heterogeneous resource
constraints at the network edge during both FL's optimization and deployment
stages. Experimental results demonstrate that RaFFM shows significant
superiority in resource utilization efficiency and uses fewer resources to
deploy FMs to FL. Despite the lower resource consumption, target models
optimized by RaFFM achieve performance on par with traditional FL methods
applied to full-sized FMs. This is evident across tasks in both natural
language processing and computer vision domains.
