  Large neural networks are very successful in various tasks. However, with
limited data, the generalization capabilities of deep neural networks are also
very limited. In this paper, we empirically start showing that intrinsically
sparse neural networks with adaptive sparse connectivity, which by design have
a strict parameter budget during the training phase, have better generalization
capabilities than their fully-connected counterparts. Besides this, we propose
a new technique to train these sparse models by combining the Sparse
Evolutionary Training (SET) procedure with neurons pruning. Operated on
MultiLayer Perceptron (MLP) and tested on 15 datasets, our proposed technique
zeros out around 50% of the hidden neurons during training, while having a
linear number of parameters to optimize with respect to the number of neurons.
The results show a competitive classification and generalization performance.
