  Batch Normalization (BatchNorm) is an extremely useful component of modern
neural network architectures, enabling optimization using higher learning rates
and achieving faster convergence. In this paper, we use mean-field theory to
analytically quantify the impact of BatchNorm on the geometry of the loss
landscape for multi-layer networks consisting of fully-connected and
convolutional layers. We show that it has a flattening effect on the loss
landscape, as quantified by the maximum eigenvalue of the Fisher Information
Matrix. These findings are then used to justify the use of larger learning
rates for networks that use BatchNorm, and we provide quantitative
characterization of the maximal allowable learning rate to ensure convergence.
Experiments support our theoretically predicted maximum learning rate, and
furthermore suggest that networks with smaller values of the BatchNorm
parameter achieve lower loss after the same number of epochs of training.
