  With the increasing number of parameters in large pre-trained models, LoRA as
a parameter-efficient fine-tuning(PEFT) method is widely used for not adding
inference overhead. The LoRA method assumes that weight changes during
fine-tuning can be approximated by low-rank matrices. However, the rank values
need to be manually verified to match different downstream tasks, and they
cannot accommodate the varying importance of different layers in the model. In
this work, we first analyze the relationship between the performance of
different layers and their ranks using SVD. Based on this, we design the
Singular-Value Based Adaptive Low-Rank Adaption(SARA), which adaptively finds
the rank during initialization by performing SVD on the pre-trained weights.
Additionally, we explore the Mixture-of-SARA(Mo-SARA), which significantly
reduces the number of parameters by fine-tuning only multiple parallel sets of
singular values controlled by a router. Extensive experiments on various
complex tasks demonstrate the simplicity and parameter efficiency of our
methods. They can effectively and adaptively find the most suitable rank for
each layer of each model.
