  The performance of a language model has been shown to be effectively modeled
as a power-law in its parameter count. Here we study the scaling behaviors of
Routing Networks: architectures that conditionally use only a subset of their
parameters while processing an input. For these models, parameter count and
computational requirement form two independent axes along which an increase
leads to better performance. In this work we derive and justify scaling laws
defined on these two variables which generalize those known for standard
language models and describe the performance of a wide range of routing
architectures trained via three different techniques. Afterwards we provide two
applications of these laws: first deriving an Effective Parameter Count along
which all models scale at the same rate, and then using the scaling
coefficients to give a quantitative comparison of the three routing techniques
considered. Our analysis derives from an extensive evaluation of Routing
Networks across five orders of magnitude of size, including models with
hundreds of experts and hundreds of billions of parameters.
