  We study randomly initialized residual networks using mean field theory and
the theory of difference equations. Classical feedforward neural networks, such
as those with tanh activations, exhibit exponential behavior on the average
when propagating inputs forward or gradients backward. The exponential forward
dynamics causes rapid collapsing of the input space geometry, while the
exponential backward dynamics causes drastic vanishing or exploding gradients.
We show, in contrast, that by adding skip connections, the network will,
depending on the nonlinearity, adopt subexponential forward and backward
dynamics, and in many cases in fact polynomial. The exponents of these
polynomials are obtained through analytic methods and proved and verified
empirically to be correct. In terms of the "edge of chaos" hypothesis, these
subexponential and polynomial laws allow residual networks to "hover over the
boundary between stability and chaos," thus preserving the geometry of the
input space and the gradient information flow. In our experiments, for each
activation function we study here, we initialize residual networks with
different hyperparameters and train them on MNIST. Remarkably, our
initialization time theory can accurately predict test time performance of
these networks, by tracking either the expected amount of gradient explosion or
the expected squared distance between the images of two input vectors.
Importantly, we show, theoretically as well as empirically, that common
initializations such as the Xavier or the He schemes are not optimal for
residual networks, because the optimal initialization variances depend on the
depth. Finally, we have made mathematical contributions by deriving several new
identities for the kernels of powers of ReLU functions by relating them to the
zeroth Bessel function of the second kind.
