  The rapid expansion of large foundation models within the pre-training and
fine-tuning framework has underscored that larger models often yield better
results. However, the scaling up of large foundation models has led to soaring
costs in fine-tuning and parameter storage, rendering extensive adaptations
impractical. This challenge has sparked the development of parameter-efficient
fine-tuning (PEFT), which focuses on optimizing a select subset of parameters
while keeping the rest fixed, significantly lowering computational and storage
overheads. While recent years have witnessed a significant success in PEFT, a
deep understanding of the fundamental principles behind these methods remains
unexplored. To this end, here we take the first step to unify all approaches by
dissecting them from a decomposition perspective. We initiate a comprehensive
mathematical analysis of these methods, allowing us to delve deeply into their
underlying mechanisms, and we explore the reasons behind the variations in
performance among different techniques. Furthermore, inspired by our
theoretical analysis, we introduce two novel PEFT methods alongside a simple
yet effective framework designed to enhance the performance of PEFT techniques
across various applications. Our empirical validations, conducted across
multiple datasets, demonstrate the efficacy of these methods, showcasing both
theoretical validity and practical performance improvements under the guidance
of our analytical findings. We believe our work will deepen researchers'
understanding of PEFT and other techniques, prompting further contemplation and
advancing the research across the whole community.
