  Modern neural network architectures often generalize well despite containing
many more parameters than the size of the training dataset. This paper explores
the generalization capabilities of neural networks trained via gradient
descent. We develop a data-dependent optimization and generalization theory
which leverages the low-rank structure of the Jacobian matrix associated with
the network. Our results help demystify why training and generalization is
easier on clean and structured datasets and harder on noisy and unstructured
datasets as well as how the network size affects the evolution of the train and
test errors during training. Specifically, we use a control knob to split the
Jacobian spectum into "information" and "nuisance" spaces associated with the
large and small singular values. We show that over the information space
learning is fast and one can quickly train a model with zero training loss that
can also generalize well. Over the nuisance space training is slower and early
stopping can help with generalization at the expense of some bias. We also show
that the overall generalization capability of the network is controlled by how
well the label vector is aligned with the information space. A key feature of
our results is that even constant width neural nets can provably generalize for
sufficiently nice datasets. We conduct various numerical experiments on deep
networks that corroborate our theoretical findings and demonstrate that: (i)
the Jacobian of typical neural networks exhibit low-rank structure with a few
large singular values and many small ones leading to a low-dimensional
information space, (ii) over the information space learning is fast and most of
the label vector falls on this space, and (iii) label noise falls on the
nuisance space and impedes optimization/generalization.
