  Modern training strategies of deep neural networks (NNs) tend to induce a
heavy-tailed (HT) spectra of layer weights. Extensive efforts to study this
phenomenon have found that NNs with HT weight spectra tend to generalize well.
A prevailing notion for the occurrence of such HT spectra attributes gradient
noise during training as a key contributing factor. Our work shows that
gradient noise is unnecessary for generating HT weight spectra: two-layer NNs
trained with full-batch Gradient Descent/Adam can exhibit HT spectra in their
weights after finite training steps. To this end, we first identify the scale
of the learning rate at which one step of full-batch Adam can lead to feature
learning in the shallow NN, particularly when learning a single index teacher
model. Next, we show that multiple optimizer steps with such (sufficiently)
large learning rates can transition the bulk of the weight's spectra into an HT
distribution. To understand this behavior, we present a novel perspective based
on the singular vectors of the weight matrices and optimizer updates. We show
that the HT weight spectrum originates from the `spike', which is generated
from feature learning and interacts with the main bulk to generate an HT
spectrum. Finally, we analyze the correlations between the HT weight spectra
and generalization after multiple optimizer updates with varying learning
rates.
