  It is widely believed that a neural network can fit a training set containing
at least as many samples as it has parameters, underpinning notions of
overparameterized and underparameterized models. In practice, however, we only
find solutions accessible via our training procedure, including the optimizer
and regularizers, limiting flexibility. Moreover, the exact parameterization of
the function class, built into an architecture, shapes its loss surface and
impacts the minima we find. In this work, we examine the ability of neural
networks to fit data in practice. Our findings indicate that: (1) standard
optimizers find minima where the model can only fit training sets with
significantly fewer samples than it has parameters; (2) convolutional networks
are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
(3) while stochastic training is thought to have a regularizing effect, SGD
actually finds minima that fit more training data than full-batch gradient
descent; (4) the difference in capacity to fit correctly labeled and
incorrectly labeled samples can be predictive of generalization; (5) ReLU
activation functions result in finding minima that fit more data despite being
designed to avoid vanishing and exploding gradients in deep architectures.
