  The sparsely gated mixture of experts (MoE) architecture sends different
inputs to different subnetworks, i.e., experts, through trainable routers. MoE
reduces the training computation significantly for large models, but its
deployment can be still memory or computation expensive for some downstream
tasks. Model pruning is a popular approach to reduce inference computation, but
its application in MoE architecture is largely unexplored. To the best of our
knowledge, this paper provides the first provably efficient technique for
pruning experts in finetuned MoE models. We theoretically prove that
prioritizing the pruning of the experts with a smaller change of the routers l2
norm from the pretrained model guarantees the preservation of test accuracy,
while significantly reducing the model size and the computational requirements.
Although our theoretical analysis is centered on binary classification tasks on
simplified MoE architecture, our expert pruning method is verified on large
vision MoE models such as VMoE and E3MoE finetuned on benchmark datasets such
as CIFAR10, CIFAR100, and ImageNet.
