  Parameter-efficient fine-tuning (PEFT) is widely studied for its
effectiveness and efficiency in the era of large language models. Low-rank
adaptation (LoRA) has demonstrated commendable performance as a popular and
representative method. However, it is implemented with a fixed intrinsic rank
that might not be the ideal setting for the downstream tasks. Recognizing the
need for more flexible downstream task adaptation, we extend the methodology of
LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA)
that enables dynamic adjustments to the intrinsic rank during the adaptation
process. First, we propose a novel method, AB-LoRA, that can effectively
estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we
gradually prune abundant and negatively impacting LoRA ranks and allocate the
pruned LoRA budgets to important Transformer modules needing higher ranks. We
have conducted experiments on various tasks, and the experimental results
demonstrate that our ALoRA method can outperform the recent baselines with
comparable tunable parameters.
