  It has been noted in existing literature that over-parameterization in ReLU
networks generally improves performance. While there could be several factors
involved behind this, we prove some desirable theoretical properties at
initialization which may be enjoyed by ReLU networks. Specifically, it is known
that He initialization in deep ReLU networks asymptotically preserves variance
of activations in the forward pass and variance of gradients in the backward
pass for infinitely wide networks, thus preserving the flow of information in
both directions. Our paper goes beyond these results and shows novel properties
that hold under He initialization: i) the norm of hidden activation of each
layer is equal to the norm of the input, and, ii) the norm of weight gradient
of each layer is equal to the product of norm of the input vector and the error
at output layer. These results are derived using the PAC analysis framework,
and hold true for finitely sized datasets such that the width of the ReLU
network only needs to be larger than a certain finite lower bound. As we show,
this lower bound depends on the depth of the network and the number of samples,
and by the virtue of being a lower bound, over-parameterized ReLU networks are
endowed with these desirable properties. For the aforementioned hidden
activation norm property under He initialization, we further extend our theory
and show that this property holds for a finite width network even when the
number of data samples is infinite. Thus we overcome several limitations of
existing papers, and show new properties of deep ReLU networks at
initialization.
