  Mixture-of-Experts (MoE) models have gained popularity in achieving
state-of-the-art performance in a wide range of tasks in computer vision and
natural language processing. They effectively expand the model capacity while
incurring a minimal increase in computation cost during training. However,
deploying such models for inference is difficult due to their large size and
complex communication pattern. In this work, we provide a characterization of
two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT)
and identify their sources of inefficiencies at deployment. We propose three
optimization techniques to mitigate sources of inefficiencies, namely (1)
Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show
that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM,
5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also
reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT.
We further propose Expert Buffering, a new caching mechanism that only keeps
hot, active experts in GPU memory while buffering the rest in CPU memory. This
reduces static memory allocation by up to 1.47$\times$. We finally propose a
load balancing methodology that provides additional scalability to the
workload.
