  Understanding when the noise in stochastic gradient descent (SGD) affects
generalization of deep neural networks remains a challenge, complicated by the
fact that networks can operate in distinct training regimes. Here we study how
the magnitude of this noise $T$ affects performance as the size of the training
set $P$ and the scale of initialization $\alpha$ are varied. For gradient
descent, $\alpha$ is a key parameter that controls if the network is
`lazy'($\alpha\gg1$) or instead learns features ($\alpha\ll1$). For
classification of MNIST and CIFAR10 images, our central results are: (i)
obtaining phase diagrams for performance in the $(\alpha,T)$ plane. They show
that SGD noise can be detrimental or instead useful depending on the training
regime. Moreover, although increasing $T$ or decreasing $\alpha$ both allow the
net to escape the lazy regime, these changes can have opposite effects on
performance. (ii) Most importantly, we find that the characteristic temperature
$T_c$ where the noise of SGD starts affecting the trained model (and eventually
performance) is a power law of $P$. We relate this finding with the observation
that key dynamical quantities, such as the total variation of weights during
training, depend on both $T$ and $P$ as power laws. These results indicate that
a key effect of SGD noise occurs late in training by affecting the stopping
process whereby all data are fitted. Indeed, we argue that due to SGD noise,
nets must develop a stronger `signal', i.e. larger informative weights, to fit
the data, leading to a longer training time. A stronger signal and a longer
training time are also required when the size of the training set $P$
increases. We confirm these views in the perceptron model, where signal and
noise can be precisely measured. Interestingly, exponents characterizing the
effect of SGD depend on the density of data near the decision boundary, as we
explain.
