  When training deep neural networks, the phenomenon of $\textit{dying
neurons}$ $\unicode{x2013}$units that become inactive or saturated, output zero
during training$\unicode{x2013}$ has traditionally been viewed as undesirable,
linked with optimization challenges, and contributing to plasticity loss in
continual learning scenarios. In this paper, we reassess this phenomenon,
focusing on sparsity and pruning. By systematically exploring the impact of
various hyperparameter configurations on dying neurons, we unveil their
potential to facilitate simple yet effective structured pruning algorithms. We
introduce $\textit{Demon Pruning}$ (DemP), a method that controls the
proliferation of dead neurons, dynamically leading to network sparsity.
Achieved through a combination of noise injection on active units and a
one-cycled schedule regularization strategy, DemP stands out for its simplicity
and broad applicability. Experiments on CIFAR10 and ImageNet datasets
demonstrate that DemP surpasses existing structured pruning techniques,
showcasing superior accuracy-sparsity tradeoffs and training speedups. These
findings suggest a novel perspective on dying neurons as a valuable resource
for efficient model compression and optimization.
