  Mixture-of-Experts (MoE) networks have been proposed as an efficient way to
scale up model capacity and implement conditional computing. However, the study
of MoE components mostly focused on the feedforward layer in Transformer
architecture. This paper proposes the Mixture of Attention Heads (MoA), a new
architecture that combines multi-head attention with the MoE mechanism. MoA
includes a set of attention heads that each has its own set of parameters.
Given an input, a router dynamically selects a subset of $k$ attention heads
per token. This conditional computation schema allows MoA to achieve stronger
performance than the standard multi-head attention layer. Furthermore, the
sparsely gated MoA can easily scale up the number of attention heads and the
number of parameters while preserving computational efficiency. In addition to
the performance improvements, MoA also automatically differentiates heads'
utilities, providing a new perspective to discuss the model's interpretability.
We conducted experiments on several important tasks, including Machine
Translation and Masked Language Modeling. Experiments have shown promising
results on several tasks against strong baselines that involve large and very
deep models.
