  Training a unified model to take multiple targets into account is a trend
towards artificial general intelligence. However, how to efficiently mitigate
the training conflicts among heterogeneous data collected from different
domains or tasks remains under-explored. In this study, we explore to leverage
Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data
training, which requires to jointly train the multiple low-rank adapters and
their shared backbone. Specifically, we introduce two variants of MoLA, namely,
MoLA-Grad and MoLA-Router, to respectively handle the target-aware and
target-agnostic scenarios during inference. The former uses task identifiers to
assign personalized low-rank adapters to each task, disentangling task-specific
knowledge towards their adapters, thereby mitigating heterogeneity conflicts.
The latter uses a novel Task-wise Decorrelation (TwD) loss to intervene the
router to learn oriented weight combinations of adapters to homogeneous tasks,
achieving similar effects. We conduct comprehensive experiments to verify the
superiority of MoLA over previous state-of-the-art methods and present in-depth
analysis on its working mechanism. Source code is available at:
https://github.com/MediaBrain-SJTU/MoLA
