  It has previously been reported that the representation that is learned in
the first layer of deep Convolutional Neural Networks (CNNs) is highly
consistent across initializations and architectures. In this work, we quantify
this consistency by considering the first layer as a filter bank and measuring
its energy distribution. We find that the energy distribution is very different
from that of the initial weights and is remarkably consistent across random
initializations, datasets, architectures and even when the CNNs are trained
with random labels. In order to explain this consistency, we derive an
analytical formula for the energy profile of linear CNNs and show that this
profile is mostly dictated by the second order statistics of image patches in
the training set and it will approach a whitening transformation when the
number of iterations goes to infinity. Finally, we show that this formula for
linear CNNs also gives an excellent fit for the energy profiles learned by
commonly used nonlinear CNNs such as ResNet and VGG, and that the first layer
of these CNNs indeed perform approximate whitening of their inputs.
