  As giant dense models advance quality but require large amounts of GPU
budgets for training, the sparsely gated Mixture-of-Experts (MoE), a kind of
conditional computation architecture, is proposed to scale models while keeping
their computation constant. Specifically, the input tokens are routed by the
gate network and only activates part of the expert network. Existing MoE
training systems only support part of mainstream MoE models (e.g. Top k)
training under expensive high-bandwidth GPU clusters. In this paper, we present
HetuMoE, a high-performance large-scale sparse MoE training system built on
Hetu. HetuMoE provides multiple gating strategies and efficient GPU kernel
implementations. To further improve the training efficiency on commodity GPU
clusters (e.g, with only 1 NiC), we introduce the hierarchical AllToAll
communication that combines hierarchical networks and aggregating messages.
Compared with existing state-of-the-art MoE systems, HetuMoE obtains at least
15% speedup. Specifically, HetuMoE outperforms DeepSpeed-MoE up to 8.1x under
the switch gate with a batch size of 32. Our code is available at:
https://github.com/PKU-DAIR/Hetu.
