  Low-rank adaptation (LoRA), as one of the most well-known representative
methods of parameter-efficient fine-tuning, freezes the backbone model and
introduces parallel adapter modules to each layer of the model. These modules
consist of two low-rank trainable matrices: a low-dimension projector (LP) and
a high-dimension projector (HP) with their product approximating the change for
updating the model weight. However, LoRA's paired LP and HP per layer limit
learned weights to specific features, ignoring the varied information extracted
by stacked layers in models like Transformers. By considering the differences
between layers and establishing connections across them when learning the
weights, we enhance the capture of relevant information for downstream tasks
using this interconnected adaptation when fine-tuning. Meanwhile, preserving
the unique characteristics of each layer and thus selectively mix the learning
traits of various layers according to a specific ratio can also be crucial in
certain tasks. In this paper, we propose Low-rank Interconnected adaptation
across layers (Lily). Specifically, we retain layer-specific LPs (local LPs)
for low-dimensional feature projection and unify all HPs into a model-wide
global HP, thereby overcoming layer-specific constraints in LoRA. The global
HP, layer-independent, supports multiple HP sub-modules, or inspired by Mixture
of Experts (MoE), HP experts capturing learning traits across all layer depths.
For the ratio to mix all the experts, we use a router inspired by MoE to
selectively adapt the features of different layers, thus obtaining a unique
expert distribution. We evaluated Lily on a wide range of downstream tasks and
achieved state-of-the-art results, outperforming LoRA and a range of
competitive methods. Code will be available at
https://github.com/blameitonme1/lily.
