  Empirical studies demonstrate that the performance of neural networks
improves with increasing number of parameters. In most of these studies, the
number of parameters is increased by increasing the network width. This begs
the question: Is the observed improvement due to the larger number of
parameters, or is it due to the larger width itself? We compare different ways
of increasing model width while keeping the number of parameters constant. We
show that for models initialized with a random, static sparsity pattern in the
weight tensors, network width is the determining factor for good performance,
while the number of weights is secondary, as long as trainability is ensured.
As a step towards understanding this effect, we analyze these models in the
framework of Gaussian Process kernels. We find that the distance between the
sparse finite-width model kernel and the infinite-width kernel at
initialization is indicative of model performance.
