  While the concept of a Sparse Neural Network has been researched for some
time, researchers have only recently made notable progress in the matter.
Techniques like Sparse Evolutionary Training allow for significantly lower
computational complexity when compared to fully connected models by reducing
redundant connections. That typically takes place in an iterative process of
weight creation and removal during network training. Although there have been
numerous approaches to optimize the redistribution of the removed weights,
there seems to be little or no study on the effect of activation functions on
the performance of the Sparse Networks. This research provides insights into
the relationship between the activation function used and the network
performance at various sparsity levels.
