  Recent theoretical works have characterized the dynamics of wide shallow
neural networks trained via gradient descent in an asymptotic mean-field limit
when the width tends towards infinity. At initialization, the random sampling
of the parameters leads to deviations from the mean-field limit dictated by the
classical Central Limit Theorem (CLT). However, since gradient descent induces
correlations among the parameters, it is of interest to analyze how these
fluctuations evolve. Here, we use a dynamical CLT to prove that the asymptotic
fluctuations around the mean limit remain bounded in mean square throughout
training. The upper bound is given by a Monte-Carlo resampling error, with a
variance that that depends on the 2-norm of the underlying measure, which also
controls the generalization error. This motivates the use of this 2-norm as a
regularization term during training. Furthermore, if the mean-field dynamics
converges to a measure that interpolates the training data, we prove that the
asymptotic deviation eventually vanishes in the CLT scaling. We also complement
these results with numerical experiments.
