  The Mixture of Experts (MoE) for language models has been proven effective in
augmenting the capacity of models by dynamically routing each input token to a
specific subset of experts for processing. Despite the success, most existing
methods face a challenge for balance between sparsity and the availability of
expert knowledge: enhancing performance through increased use of expert
knowledge often results in diminishing sparsity during expert selection. To
mitigate this contradiction, we propose HyperMoE, a novel MoE framework built
upon Hypernetworks. This framework integrates the computational processes of
MoE with the concept of knowledge transferring in multi-task learning. Specific
modules generated based on the information of unselected experts serve as
supplementary information, which allows the knowledge of experts not selected
to be used while maintaining selection sparsity. Our comprehensive empirical
evaluations across multiple datasets and backbones establish that HyperMoE
significantly outperforms existing MoE methods under identical conditions
concerning the number of experts.
