  We analyze the dynamics of finite width effects in wide but finite feature
learning neural networks. Starting from a dynamical mean field theory
description of infinite width deep neural network kernel and prediction
dynamics, we provide a characterization of the $O(1/\sqrt{\text{width}})$
fluctuations of the DMFT order parameters over random initializations of the
network weights. Our results, while perturbative in width, unlike prior
analyses, are non-perturbative in the strength of feature learning. In the lazy
limit of network training, all kernels are random but static in time and the
prediction variance has a universal form. However, in the rich, feature
learning regime, the fluctuations of the kernels and predictions are
dynamically coupled with a variance that can be computed self-consistently. In
two layer networks, we show how feature learning can dynamically reduce the
variance of the final tangent kernel and final network predictions. We also
show how initialization variance can slow down online learning in wide but
finite networks. In deeper networks, kernel variance can dramatically
accumulate through subsequent layers at large feature learning strengths, but
feature learning continues to improve the signal-to-noise ratio of the feature
kernels. In discrete time, we demonstrate that large learning rate phenomena
such as edge of stability effects can be well captured by infinite width
dynamics and that initialization variance can decrease dynamically. For CNNs
trained on CIFAR-10, we empirically find significant corrections to both the
bias and variance of network dynamics due to finite width.
