  Current methods for pruning neural network weights iteratively apply
magnitude-based pruning on the model weights and re-train the resulting model
to recover lost accuracy. In this work, we show that such strategies do not
allow for the recovery of erroneously pruned weights. To enable weight
recovery, we propose a simple strategy called \textit{cyclical pruning} which
requires the pruning schedule to be periodic and allows for weights pruned
erroneously in one cycle to recover in subsequent ones. Experimental results on
both linear models and large-scale deep neural networks show that cyclical
pruning outperforms existing pruning algorithms, especially at high sparsity
ratios. Our approach is easy to tune and can be readily incorporated into
existing pruning pipelines to boost performance.
