  Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for
pretrained deep neural networks have captured widespread interest. In this
work, we study the enhancement of current PEFT methods by incorporating the
spectral information of pretrained weight matrices into the fine-tuning
procedure. We investigate two spectral adaptation mechanisms, namely additive
tuning and orthogonal rotation of the top singular vectors, both are done via
first carrying out Singular Value Decomposition (SVD) of pretrained weights and
then fine-tuning the top spectral space. We provide a theoretical analysis of
spectral fine-tuning and show that our approach improves the rank capacity of
low-rank adapters given a fixed trainable parameter budget. We show through
extensive experiments that the proposed fine-tuning model enables better
parameter efficiency and tuning performance as well as benefits multi-adapter
fusion. The code will be open-sourced for reproducibility.
