  Sparse neural networks have shown similar or better generalization
performance than their dense counterparts while having higher parameter
efficiency. This has motivated a number of works to learn or search for high
performing sparse networks. While reports of task performance or efficiency
gains are impressive, standard baselines are lacking leading to poor
comparability and unreliable reproducibility across methods. In this work, we
propose Random Search as a baseline algorithm for finding good sparse
configurations and study its performance. We apply Random Search on the node
space of an overparameterized network with the goal of finding better
initialized sparse sub-networks that are positioned more advantageously in the
loss landscape. We record the post-training performances of the found sparse
networks and at various levels of sparsity, and compare against both their
fully connected parent networks and random sparse configurations at the same
sparsity levels. First, we demonstrate performance at different levels of
sparsity and highlight that a significant level of performance can still be
preserved even when the network is highly sparse. Second, we observe that for
this sparse architecture search task, initialized sparse networks found by
Random Search neither perform better nor converge more efficiently than their
random counterparts. Thus we conclude that Random Search may be viewed as a
reasonable neutral baseline for sparsity search methods.
