  The application of mixture-of-experts (MoE) is gaining popularity due to its
ability to improve model's performance. In an MoE structure, the gate layer
plays a significant role in distinguishing and routing input features to
different experts. This enables each expert to specialize in processing their
corresponding sub-tasks. However, the gate's routing mechanism also gives rise
to narrow vision: the individual MoE's expert fails to use more samples in
learning the allocated sub-task, which in turn limits the MoE to further
improve its generalization ability. To effectively address this, we propose a
method called Mixture-of-Distilled-Expert (MoDE), which applies moderate mutual
distillation among experts to enable each expert to pick up more features
learned by other experts and gain more accurate perceptions on their original
allocated sub-tasks. We conduct plenty experiments including tabular, NLP and
CV datasets, which shows MoDE's effectiveness, universality and robustness.
Furthermore, we develop a parallel study through innovatively constructing
"expert probing", to experimentally prove why MoDE works: moderate distilling
knowledge can improve each individual expert's test performances on their
assigned tasks, leading to MoE's overall performance improvement.
