  Mixture-of-experts (MoE) is becoming popular due to its success in improving
the model quality, especially in Transformers. By routing tokens with a sparse
gate to a few experts (i.e., a small pieces of the full model), MoE can easily
increase the model parameters to a very large scale while keeping the
computation cost in a constant level. Most existing works just initialize some
random experts, set a fixed gating strategy (e.g., Top-k), and train the model
from scratch in an ad-hoc way. We identify that these MoE models are suffering
from the immature experts and unstable sparse gate, which are harmful to the
convergence performance. In this paper, we propose an efficient end-to-end MoE
training framework called EvoMoE. EvoMoE starts from training one single expert
and gradually evolves into a large and sparse MoE structure. EvoMoE mainly
contains two phases: the expert-diversify phase to train the base expert for a
while and spawn multiple diverse experts from it, and the gate-sparsify phase
to learn an adaptive sparse gate and activate a dynamic number of experts.
EvoMoE naturally decouples the joint learning of both the experts and the
sparse gate and focuses on learning the basic knowledge with a single expert at
the early training stage. Then it diversifies the experts and continues to
train the MoE with a novel Dense-to-Sparse gate (DTS-Gate). Specifically,
instead of using a permanent sparse gate, DTS-Gate begins as a dense gate that
routes tokens to all experts, then gradually and adaptively becomes sparser
while routes to fewer experts. Evaluations are conducted on three popular
models and tasks, including RoBERTa for masked language modeling task, GPT for
language modeling task and Transformer for machine translation task. The
results show that EvoMoE outperforms existing baselines, including Switch, BASE
Layer, Hash Layer and StableMoE.
