  Instruction tuning of Large Vision-language Models (LVLMs) has revolutionized
the development of versatile models with zero-shot generalization across a wide
range of downstream vision-language tasks. However, the diversity of training
tasks of different sources and formats would lead to inevitable task conflicts,
where different tasks conflict for the same set of model parameters, resulting
in sub-optimal instruction-following abilities. To address that, we propose the
Mixture of Cluster-conditional LoRA Experts (MoCLE), a novel Mixture of Experts
(MoE) architecture designed to activate the task-customized model parameters
based on the instruction clusters. A separate universal expert is further
incorporated to improve generalization capabilities of MoCLE for novel
instructions. Extensive experiments on InstructBLIP and LLaVA demonstrate the
effectiveness of MoCLE.
