  Large pre-trained models have exhibited remarkable achievements across
various domains. The substantial training costs associated with these models
have led to wide studies of fine-tuning for effectively harnessing their
capabilities in solving downstream tasks. Yet, conventional fine-tuning
approaches become infeasible when the model lacks access to downstream data due
to privacy concerns. Naively integrating fine-tuning approaches with the
emerging federated learning frameworks incurs substantial communication
overhead and exerts high demand on local computing resources, making it
impractical for common resource-limited devices. In this paper, we introduce
SFPrompt, an innovative privacy-preserving fine-tuning method tailored for the
federated setting where direct uploading of raw data is prohibited and local
devices are resource-constrained to run a complete pre-trained model. In
essence, SFPrompt judiciously combines split learning with federated learning
to handle these challenges. Specifically, the pre-trained model is first
partitioned into client and server components, thereby streamlining the
client-side model and substantially alleviating computational demands on local
resources. SFPrompt then introduces soft prompts into the federated model to
enhance the fine-tuning performance. To further reduce communication costs, a
novel dataset pruning algorithm and a local-loss update strategy are devised
during the fine-tuning process. Extensive experiments demonstrate that SFPrompt
delivers competitive performance as the federated full fine-tuning approach
while consuming a mere 0.46% of local computing resources and incurring 53%
less communication cost.
