  How does the geometric representation of a dataset change after the
application of each randomly initialized layer of a neural network? The
celebrated Johnson--Lindenstrauss lemma answers this question for linear
fully-connected neural networks (FNNs), stating that the geometry is
essentially preserved. For FNNs with the ReLU activation, the angle between two
inputs contracts according to a known mapping. The question for non-linear
convolutional neural networks (CNNs) becomes much more intricate. To answer
this question, we introduce a geometric framework. For linear CNNs, we show
that the Johnson--Lindenstrauss lemma continues to hold, namely, that the angle
between two inputs is preserved. For CNNs with ReLU activation, on the other
hand, the behavior is richer: The angle between the outputs contracts, where
the level of contraction depends on the nature of the inputs. In particular,
after one layer, the geometry of natural images is essentially preserved,
whereas for Gaussian correlated inputs, CNNs exhibit the same contracting
behavior as FNNs with ReLU activation.
