  Mixture-of-experts (MoE) models that employ sparse activation have
demonstrated effectiveness in significantly increasing the number of parameters
while maintaining low computational requirements per token. However, recent
studies have established that MoE models are inherently parameter-inefficient
as the improvement in performance diminishes with an increasing number of
experts. We hypothesize this parameter inefficiency is a result of all experts
having equal capacity, which may not adequately meet the varying complexity
requirements of different tokens or tasks. In light of this, we propose
Stratified Mixture of Experts (SMoE) models, which feature a stratified
structure and can assign dynamic capacity to different tokens. We demonstrate
the effectiveness of SMoE on three multilingual machine translation benchmarks,
containing 4, 15, and 94 language pairs, respectively. We show that SMoE
outperforms multiple state-of-the-art MoE models with the same or fewer
parameters.
