  We rigorously prove a central limit theorem for neural network models with a
single hidden layer. The central limit theorem is proven in the asymptotic
regime of simultaneously (A) large numbers of hidden units and (B) large
numbers of stochastic gradient descent training iterations. Our result
describes the neural network's fluctuations around its mean-field limit. The
fluctuations have a Gaussian distribution and satisfy a stochastic partial
differential equation. The proof relies upon weak convergence methods from
stochastic analysis. In particular, we prove relative compactness for the
sequence of processes and uniqueness of the limiting process in a suitable
Sobolev space.
