  Recently, Mixture of Experts (MoE) Transformers have garnered increasing
attention due to their advantages in model capacity and computational
efficiency. However, studies have indicated that MoE Transformers underperform
vanilla Transformers in many downstream tasks, significantly diminishing the
practical value of MoE models. To explain this issue, we propose that the
pre-training performance and transfer capability of a model are joint
determinants of its downstream task performance. MoE models, in comparison to
vanilla models, have poorer transfer capability, leading to their subpar
performance in downstream tasks. To address this issue, we introduce the
concept of transfer capability distillation, positing that although vanilla
models have weaker performance, they are effective teachers of transfer
capability. The MoE models guided by vanilla models can achieve both strong
pre-training performance and transfer capability, ultimately enhancing their
performance in downstream tasks. We design a specific distillation method and
conduct experiments on the BERT architecture. Experimental results show a
significant improvement in downstream performance of MoE models, and many
further evidences also strongly support the concept of transfer capability
distillation. Finally, we attempt to interpret transfer capability distillation
and provide some insights from the perspective of model feature.
