  LoRA has gained widespread acceptance in the fine-tuning of large pre-trained
models to cater to a diverse array of downstream tasks, showcasing notable
effectiveness and efficiency, thereby solidifying its position as one of the
most prevalent fine-tuning techniques. Due to the modular nature of LoRA's
plug-and-play plugins, researchers have delved into the amalgamation of
multiple LoRAs to empower models to excel across various downstream tasks.
Nonetheless, extant approaches for LoRA fusion grapple with inherent
challenges. Direct arithmetic merging may result in the loss of the original
pre-trained model's generative capabilities or the distinct identity of LoRAs,
thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based
fusion exhibits limitations concerning the requisite flexibility for the
effective combination of multiple LoRAs. In response to these challenges, this
paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses
hierarchical control and unfettered branch selection. The MoLE approach not
only achieves superior LoRA fusion performance in comparison to direct
arithmetic merging but also retains the crucial flexibility for combining LoRAs
effectively. Extensive experimental evaluations conducted in both the Natural
Language Processing (NLP) and Vision & Language (V&L) domains substantiate the
efficacy of MoLE.
