  We investigate the spectral properties of linear-width feed-forward neural
networks, where the sample size is asymptotically proportional to network
width. Empirically, we show that the spectra of weight in this high dimensional
regime are invariant when trained by gradient descent for small constant
learning rates; we provide a theoretical justification for this observation and
prove the invariance of the bulk spectra for both conjugate and neural tangent
kernels. We demonstrate similar characteristics when training with stochastic
gradient descent with small learning rates. When the learning rate is large, we
exhibit the emergence of an outlier whose corresponding eigenvector is aligned
with the training data structure. We also show that after adaptive gradient
training, where a lower test error and feature learning emerge, both weight and
kernel matrices exhibit heavy tail behavior. Simple examples are provided to
explain when heavy tails can have better generalizations. We exhibit different
spectral properties such as invariant bulk, spike, and heavy-tailed
distribution from a two-layer neural network using different training
strategies, and then correlate them to the feature learning. Analogous
phenomena also appear when we train conventional neural networks with
real-world data. We conclude that monitoring the evolution of the spectra
during training is an essential step toward understanding the training dynamics
and feature learning.
