  This article considers fully connected neural networks with Gaussian random
weights and biases as well as $L$ hidden layers, each of width proportional to
a large parameter $n$. For polynomially bounded non-linearities we give sharp
estimates in powers of $1/n$ for the joint cumulants of the network output and
its derivatives. Moreover, we show that network cumulants form a perturbatively
solvable hierarchy in powers of $1/n$ in that $k$-th order cumulants in one
layer have recursions that depend to leading order in $1/n$ only on $j$-th
order cumulants at the previous layer with $j\leq k$. By solving a variety of
such recursions, however, we find that the depth-to-width ratio $L/n$ plays the
role of an effective network depth, controlling both the scale of fluctuations
at individual neurons and the size of inter-neuron correlations. Thus, while
the cumulant recursions we derive form a hierarchy in powers of $1/n$,
contributions of order $1/n^k$ often grow like $L^k$ and are hence
non-negligible at positive $L/n$. We use this to study a somewhat simplified
version of the exploding and vanishing gradient problem, proving that this
particular variant occurs if and only if $L/n$ is large. Several key ideas in
this article were first developed at a physics level of rigor in a recent
monograph of Daniel A. Roberts, Sho Yaida, and the author. This article not
only makes these ideas mathematically precise but also significantly extends
them, opening the way to obtaining corrections to all orders in $1/n$.
