  We argue that many properties of fully-connected feedforward neural networks
(FCNNs), also called multi-layer perceptrons (MLPs), are explainable from the
analysis of a single pair of operations, namely a random projection into a
higher-dimensional space than the input, followed by a sparsification
operation. For convenience, we call this pair of successive operations
expand-and-sparsify following the terminology of Dasgupta. We show how
expand-and-sparsify can explain the observed phenomena that have been discussed
in the literature, such as the so-called Lottery Ticket Hypothesis, the
surprisingly good performance of randomly-initialized untrained neural
networks, the efficacy of Dropout in training and most importantly, the
mysterious generalization ability of overparameterized models, first
highlighted by Zhang et al. and subsequently identified even in non-neural
network models by Belkin et al.
