  When the parameters are independently and identically distributed
(initialized) neural networks exhibit undesirable properties that emerge as the
number of layers increases, e.g. a vanishing dependency on the input and a
concentration on restrictive families of functions including constant
functions. We consider parameter distributions that shrink as the number of
layers increases in order to recover well-behaved stochastic processes in the
limit of infinite depth. This leads to set forth a link between infinitely deep
residual networks and solutions to stochastic differential equations, i.e.
diffusion processes. We show that these limiting processes do not suffer from
the aforementioned issues and investigate their properties.
