  Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
Language Models (LLMs) with constrained resources. Although there have been
various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
LLMs is still underexplored. In this work, we study the PEFT method for LLMs
with the Mixture-of-Experts (MoE) architecture and the contents of this work
are mainly threefold: (1) We investigate the dispersion degree of the activated
experts in customized tasks, and found that the routing distribution for a
specific task tends to be highly concentrated, while the distribution of
activated experts varies significantly across different tasks. (2) We propose
Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
to downstream tasks while freezing the other experts and modules; experimental
results demonstrate that our method not only improves the tuning efficiency,
but also matches or even surpasses the performance of full-parameter
fine-tuning. (3) We further analyze the impact of the MoE architecture on
expert-specialized fine-tuning. We find that MoE models with finer-grained
experts are more advantageous in selecting the combination of experts that are
most relevant to downstream tasks, thereby enhancing both the training
efficiency and effectiveness. Our code is available at
https://github.com/deepseek-ai/ESFT.
