  Sparse Neural Networks (SNNs) can potentially demonstrate similar performance
to their dense counterparts while saving significant energy and memory at
inference. However, the accuracy drop incurred by SNNs, especially at high
pruning ratios, can be an issue in critical deployment conditions. While recent
works mitigate this issue through sophisticated pruning techniques, we shift
our focus to an overlooked factor: hyperparameters and activation functions.
Our analyses have shown that the accuracy drop can additionally be attributed
to (i) Using ReLU as the default choice for activation functions unanimously,
and (ii) Fine-tuning SNNs with the same hyperparameters as dense counterparts.
Thus, we focus on learning a novel way to tune activation functions for sparse
networks and combining these with a separate hyperparameter optimization (HPO)
regime for sparse networks. By conducting experiments on popular DNN models
(LeNet-5, VGG-16, ResNet-18, and EfficientNet-B0) trained on MNIST, CIFAR-10,
and ImageNet-16 datasets, we show that the novel combination of these two
approaches, dubbed Sparse Activation Function Search, short: SAFS, results in
up to 15.53%, 8.88%, and 6.33% absolute improvement in the accuracy for
LeNet-5, VGG-16, and ResNet-18 over the default training protocols, especially
at high pruning ratios. Our code can be found at https://github.com/automl/SAFS
