  We study whether a neural network optimizes to the same, linearly connected
minimum under different samples of SGD noise (e.g., random data order and
augmentation). We find that standard vision models become stable to SGD noise
in this way early in training. From then on, the outcome of optimization is
determined to a linearly connected region. We use this technique to study
iterative magnitude pruning (IMP), the procedure used by work on the lottery
ticket hypothesis to identify subnetworks that could have trained in isolation
to full accuracy. We find that these subnetworks only reach full accuracy when
they are stable to SGD noise, which either occurs at initialization for
small-scale settings (MNIST) or early in training for large-scale settings
(ResNet-50 and Inception-v3 on ImageNet).
