  In this work, we build a generic architecture of Convolutional Neural
Networks to discover empirical properties of neural networks. Our first
contribution is to introduce a state-of-the-art framework that depends upon few
hyper parameters and to study the network when we vary them. It has no max
pooling, no biases, only 13 layers, is purely convolutional and yields up to
95.4% and 79.6% accuracy respectively on CIFAR10 and CIFAR100. We show that the
nonlinearity of a deep network does not need to be continuous, non expansive or
point-wise, to achieve good performance. We show that increasing the width of
our network permits being competitive with very deep networks. Our second
contribution is an analysis of the contraction and separation properties of
this network. Indeed, a 1-nearest neighbor classifier applied on deep features
progressively improves with depth, which indicates that the representation is
progressively more regular. Besides, we defined and analyzed local support
vectors that separate classes locally. All our experiments are reproducible and
code is available online, based on TensorFlow.
