  The rectified linear unit (ReLU) is a highly successful activation function
in neural networks as it allows networks to easily obtain sparse
representations, which reduces overfitting in overparameterized networks.
However, in network pruning, we find that the sparsity introduced by ReLU,
which we quantify by a term called dynamic dead neuron rate (DNR), is not
beneficial for the pruned network. Interestingly, the more the network is
pruned, the smaller the dynamic DNR becomes during optimization. This motivates
us to propose a method to explicitly reduce the dynamic DNR for the pruned
network, i.e., de-sparsify the network. We refer to our method as
Activating-while-Pruning (AP). We note that AP does not function as a
stand-alone method, as it does not evaluate the importance of weights. Instead,
it works in tandem with existing pruning methods and aims to improve their
performance by selective activation of nodes to reduce the dynamic DNR. We
conduct extensive experiments using popular networks (e.g., ResNet, VGG) via
two classical and three state-of-the-art pruning methods. The experimental
results on public datasets (e.g., CIFAR-10/100) suggest that AP works well with
existing pruning methods and improves the performance by 3% - 4%. For larger
scale datasets (e.g., ImageNet) and state-of-the-art networks (e.g., vision
transformer), we observe an improvement of 2% - 3% with AP as opposed to
without. Lastly, we conduct an ablation study to examine the effectiveness of
the components comprising AP.
