  The population loss of trained deep neural networks often follows precise
power-law scaling relations with either the size of the training dataset or the
number of parameters in the network. We propose a theory that explains the
origins of and connects these scaling laws. We identify variance-limited and
resolution-limited scaling behavior for both dataset and model size, for a
total of four scaling regimes. The variance-limited scaling follows simply from
the existence of a well-behaved infinite data or infinite width limit, while
the resolution-limited regime can be explained by positing that models are
effectively resolving a smooth data manifold. In the large width limit, this
can be equivalently obtained from the spectrum of certain kernels, and we
present evidence that large width and large dataset resolution-limited scaling
exponents are related by a duality. We exhibit all four scaling regimes in the
controlled setting of large random feature and pretrained models and test the
predictions empirically on a range of standard architectures and datasets. We
also observe several empirical relationships between datasets and scaling
exponents under modifications of task and architecture aspect ratio. Our work
provides a taxonomy for classifying different scaling regimes, underscores that
there can be different mechanisms driving improvements in loss, and lends
insight into the microscopic origins of and relationships between scaling
exponents.
