  The recent surge in Large Language Models (LLMs) has garnered significant
attention across numerous fields. Fine-tuning is often required to fit general
LLMs for a specific domain, like the web-based healthcare system. However, two
problems arise during fine-tuning LLMs for medical applications. One is the
task variety problem, which involves distinct tasks in real-world medical
scenarios. The variety often leads to sub-optimal fine-tuning for data
imbalance and seesaw problems. Besides, the large amount of parameters in LLMs
leads to huge time and computation consumption by fine-tuning. To address these
two problems, we propose a novel parameter efficient fine-tuning framework for
multi-task medical applications, dubbed as MOELoRA. The designed framework aims
to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning
and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For
unifying MOE and LoRA, we devise multiple experts as the trainable parameters,
where each expert consists of a pair of low-rank matrices to retain the small
size of trainable parameters. Then, a task-motivated gate function for all
MOELoRA layers is proposed, which can control the contributions of each expert
and produce distinct parameters for various tasks. We conduct experiments on a
multi-task medical dataset, indicating MOELoRA outperforms the existing
parameter efficient fine-tuning methods. The code is available online.
