  A central question of machine learning is how deep nets manage to learn tasks
in high dimensions. An appealing hypothesis is that they achieve this feat by
building a representation of the data where information irrelevant to the task
is lost. For image datasets, this view is supported by the observation that
after (and not before) training, the neural representation becomes less and
less sensitive to diffeomorphisms acting on images as the signal propagates
through the net. This loss of sensitivity correlates with performance, and
surprisingly correlates with a gain of sensitivity to white noise acquired
during training. These facts are unexplained, and as we demonstrate still hold
when white noise is added to the images of the training set. Here, we (i) show
empirically for various architectures that stability to image diffeomorphisms
is achieved by both spatial and channel pooling, (ii) introduce a model
scale-detection task which reproduces our empirical observations on spatial
pooling and (iii) compute analitically how the sensitivity to diffeomorphisms
and noise scales with depth due to spatial pooling. The scalings are found to
depend on the presence of strides in the net architecture. We find that the
increased sensitivity to noise is due to the perturbing noise piling up during
pooling, after being rectified by ReLU units.
