  A recent line of works studied wide deep neural networks (DNNs) by
approximating them as Gaussian Processes (GPs). A DNN trained with gradient
flow was shown to map to a GP governed by the Neural Tangent Kernel (NTK),
whereas earlier works showed that a DNN with an i.i.d. prior over its weights
maps to the so-called Neural Network Gaussian Process (NNGP). Here we consider
a DNN training protocol, involving noise, weight decay and finite width, whose
outcome corresponds to a certain non-Gaussian stochastic process. An analytical
framework is then introduced to analyze this non-Gaussian process, whose
deviation from a GP is controlled by the finite width. Our contribution is
three-fold: (i) In the infinite width limit, we establish a correspondence
between DNNs trained with noisy gradients and the NNGP, not the NTK. (ii) We
provide a general analytical form for the finite width correction (FWC) for
DNNs with arbitrary activation functions and depth and use it to predict the
outputs of empirical finite networks with high accuracy. Analyzing the FWC
behavior as a function of $n$, the training set size, we find that it is
negligible for both the very small $n$ regime, and, surprisingly, for the large
$n$ regime (where the GP error scales as $O(1/n)$). (iii) We flesh out
algebraically how these FWCs can improve the performance of finite
convolutional neural networks (CNNs) relative to their GP counterparts on image
classification tasks.
