  Deep neural networks (DNN) are the state of the art on many engineering
problems such as computer vision and audition. A key factor in the success of
the DNN is scalability - bigger networks work better. However, the reason for
this scalability is not yet well understood. Here, we interpret the DNN as a
discrete system, of linear filters followed by nonlinear activations, that is
subject to the laws of sampling theory. In this context, we demonstrate that
over-sampled networks are more selective, learn faster and learn more robustly.
Our findings may ultimately generalize to the human brain.
