  We establish novel rates for the Gaussian approximation of random deep neural
networks with Gaussian parameters (weights and biases) and Lipschitz activation
functions, in the wide limit. Our bounds apply for the joint output of a
network evaluated any finite input set, provided a certain non-degeneracy
condition of the infinite-width covariances holds. We demonstrate that the
distance between the network output and the corresponding Gaussian
approximation scales inversely with the width of the network, exhibiting faster
convergence than the naive heuristic suggested by the central limit theorem. We
also apply our bounds to obtain theoretical approximations for the exact
Bayesian posterior distribution of the network, when the likelihood is a
bounded Lipschitz function of the network output evaluated on a (finite)
training set. This includes popular cases such as the Gaussian likelihood, i.e.
exponential of minus the mean squared error.
