  We study the distribution of a fully connected neural network with random
Gaussian weights and biases in which the hidden layer widths are proportional
to a large constant $n$. Under mild assumptions on the non-linearity, we obtain
quantitative bounds on normal approximations valid at large but finite $n$ and
any fixed network depth. Our theorems show both for the finite-dimensional
distributions and the entire process, that the distance between a random fully
connected network (and its derivatives) to the corresponding infinite width
Gaussian process scales like $n^{-\gamma}$ for $\gamma>0$, with the exponent
depending on the metric used to measure discrepancy. Our bounds are strictly
stronger in terms of their dependence on network width than any previously
available in the literature; in the one-dimensional case, we also prove that
they are optimal, i.e., we establish matching lower bounds.
