  The Mixture-of-Experts (MoE) technique can scale up the model size of
Transformers with an affordable computational overhead. We point out that
existing learning-to-route MoE methods suffer from the routing fluctuation
issue, i.e., the target expert of the same input may change along with
training, but only one expert will be activated for the input during inference.
The routing fluctuation tends to harm sample efficiency because the same input
updates different experts but only one is finally used. In this paper, we
propose StableMoE with two training stages to address the routing fluctuation
problem. In the first training stage, we learn a balanced and cohesive routing
strategy and distill it into a lightweight router decoupled from the backbone
model. In the second training stage, we utilize the distilled router to
determine the token-to-expert assignment and freeze it for a stable routing
strategy. We validate our method on language modeling and multilingual machine
translation. The results show that StableMoE outperforms existing MoE methods
in terms of both convergence speed and performance.
