  Recent studies have shown that many important aspects of neural network
learning take place within the very earliest iterations or epochs of training.
For example, sparse, trainable sub-networks emerge (Frankle et al., 2019),
gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the
network undergoes a critical period (Achille et al., 2019). Here, we examine
the changes that deep neural networks undergo during this early phase of
training. We perform extensive measurements of the network state during these
early iterations of training and leverage the framework of Frankle et al.
(2019) to quantitatively probe the weight distribution and its reliance on
various aspects of the dataset. We find that, within this framework, deep
networks are not robust to reinitializing with random weights while maintaining
signs, and that weight distributions are highly non-independent even after only
a few hundred iterations. Despite this behavior, pre-training with blurred
inputs or an auxiliary self-supervised task can approximate the changes in
supervised networks, suggesting that these changes are not inherently
label-dependent, though labels significantly accelerate this process. Together,
these results help to elucidate the network changes occurring during this
pivotal initial period of learning.
