  Large pre-trained models, such as large language models (LLMs), present
significant resource challenges for fine-tuning due to their extensive
parameter sizes, especially for applications in mobile systems. To address
this, Low-Rank Adaptation (LoRA) has been developed to reduce resource
consumption while maintaining satisfactory fine-tuning results. Despite its
effectiveness, the original LoRA method faces challenges of suboptimal
performance and overfitting. This paper investigates the intrinsic dimension of
the matrix updates approximated by the LoRA method and reveals the performance
benefits of increasing this intrinsic dimension. By employing regularization
and a gradient masking method that encourages higher intrinsic dimension, the
proposed method, termed Regularized and Masked LoRA (RM-LoRA), achieves
superior generalization performance with the same or lower trainable parameter
budget compared to the original LoRA and its latest variants across various
open-source vision and language datasets.
