  Batch normalization dramatically increases the largest trainable depth of
residual networks, and this benefit has been crucial to the empirical success
of deep residual networks on a wide range of benchmarks. We show that this key
benefit arises because, at initialization, batch normalization downscales the
residual branch relative to the skip connection, by a normalizing factor on the
order of the square root of the network depth. This ensures that, early in
training, the function computed by normalized residual blocks in deep networks
is close to the identity function (on average). We use this insight to develop
a simple initialization scheme that can train deep residual networks without
normalization. We also provide a detailed empirical study of residual networks,
which clarifies that, although batch normalized networks can be trained with
larger learning rates, this effect is only beneficial in specific compute
regimes, and has minimal benefits when the batch size is small.
