  Parameter-efficient fine-tuning (PEFT) has emerged as the predominant
technique for fine-tuning in the era of large language models. However,
existing PEFT methods still have inadequate training efficiency. Firstly, the
utilization of large-scale foundation models during the training process is
excessively redundant for certain fine-tuning tasks. Secondly, as the model
size increases, the growth in trainable parameters of empirically added PEFT
modules becomes non-negligible and redundant, leading to inefficiency. To
achieve task-specific efficient fine-tuning, we propose the Light-PEFT
framework, which includes two methods: Masked Early Pruning of the Foundation
Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework
allows for the simultaneous estimation of redundant parameters in both the
foundation model and PEFT modules during the early stage of training. These
parameters can then be pruned for more efficient fine-tuning. We validate our
approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT,
parameters of the foundation model can be pruned by up to over 40%, while still
controlling trainable parameters to be only 25% of the original PEFT method.
Compared to utilizing the PEFT method directly, Light-PEFT achieves training
and inference speedup, reduces memory usage, and maintains comparable
performance and the plug-and-play feature of PEFT.
