  With increasing privacy concerns on data, recent studies have made
significant progress using federated learning (FL) on privacy-sensitive natural
language processing (NLP) tasks. Much literature suggests fully fine-tuning
pre-trained language models (PLMs) in the FL paradigm can mitigate the data
heterogeneity problem and close the performance gap with centralized training.
However, large PLMs bring the curse of prohibitive communication overhead and
local model adaptation costs for the FL system. To this end, we introduce
various parameter-efficient tuning (PETuning) methods into federated learning.
Specifically, we provide a holistic empirical study of representative PLMs
tuning methods in FL. The experimental results cover the analysis of data
heterogeneity levels, data scales, and different FL scenarios. Overall
communication overhead can be significantly reduced by locally tuning and
globally aggregating lightweight model parameters while maintaining acceptable
performance in various FL settings. To facilitate the research of PETuning in
FL, we also develop a federated tuning framework FedPETuning, which allows
practitioners to exploit different PETuning methods under the FL training
paradigm conveniently. The source code is available at
\url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.
