  Neural networks are powerful functions with widespread use, but the
theoretical behaviour of these functions is not fully understood. Creating deep
neural networks by stacking many layers has achieved exceptional performance in
many applications and contributed to the recent explosion of these methods.
Previous works have shown that depth can exponentially increase the
expressibility of the network. However, as networks get deeper and deeper, they
are more susceptible to becoming degenerate. We observe this degeneracy in the
sense that on initialization, inputs tend to become more and more correlated as
they travel through the layers of the network. If a network has too many
layers, it tends to approximate a (random) constant function, making it
effectively incapable of distinguishing between inputs. This seems to affect
the training of the network and cause it to perform poorly, as we empirically
investigate in this paper. We use a simple algorithm that can accurately
predict the level of degeneracy for any given fully connected ReLU network
architecture, and demonstrate how the predicted degeneracy relates to training
dynamics of the network. We also compare this prediction to predictions derived
using infinite width networks.
