  Mean field theory is widely used in the theoretical studies of neural
networks. In this paper, we analyze the role of depth in the concentration of
mean-field predictions, specifically for deep multilayer perceptron (MLP) with
batch normalization (BN) at initialization. By scaling the network width to
infinity, it is postulated that the mean-field predictions suffer from
layer-wise errors that amplify with depth. We demonstrate that BN stabilizes
the distribution of representations that avoids the error propagation of
mean-field predictions. This stabilization, which is characterized by a
geometric mixing property, allows us to establish concentration bounds for mean
field predictions in infinitely-deep neural networks with a finite width.
