  Batch normalization (BN) is a technique to normalize activations in
intermediate layers of deep neural networks. Its tendency to improve accuracy
and speed up training have established BN as a favorite technique in deep
learning. Yet, despite its enormous success, there remains little consensus on
the exact reason and mechanism behind these improvements. In this paper we take
a step towards a better understanding of BN, following an empirical approach.
We conduct several experiments, and show that BN primarily enables training
with larger learning rates, which is the cause for faster convergence and
better generalization. For networks without BN we demonstrate how large
gradient updates can result in diverging loss and activations growing
uncontrollably with network depth, which limits possible learning rates. BN
avoids this problem by constantly correcting activations to be zero-mean and of
unit standard deviation, which enables larger gradient steps, yields faster
convergence and may help bypass sharp local minima. We further show various
ways in which gradients and activations of deep unnormalized networks are
ill-behaved. We contrast our results against recent findings in random matrix
theory, shedding new light on classical initialization schemes and their
consequences.
