  Sparse expert models are a thirty-year old concept re-emerging as a popular
architecture in deep learning. This class of architecture encompasses
Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and
others, all with the unifying idea that each example is acted on by a subset of
the parameters. By doing so, the degree of sparsity decouples the parameter
count from the compute per example allowing for extremely large, but efficient
models. The resulting models have demonstrated significant improvements across
diverse domains such as natural language processing, computer vision, and
speech recognition. We review the concept of sparse expert models, provide a
basic description of the common algorithms, contextualize the advances in the
deep learning era, and conclude by highlighting areas for future work.
