  Network pruning is widely used to compress Deep Neural Networks (DNNs). The
Soft Filter Pruning (SFP) method zeroizes the pruned filters during training
while updating them in the next training epoch. Thus the trained information of
the pruned filters is completely dropped. To utilize the trained pruned
filters, we proposed a SofteR Filter Pruning (SRFP) method and its variant,
Asymptotic SofteR Filter Pruning (ASRFP), simply decaying the pruned weights
with a monotonic decreasing parameter. Our methods perform well across various
networks, datasets and pruning rates, also transferable to weight pruning. On
ILSVRC-2012, ASRFP prunes 40% of the parameters on ResNet-34 with 1.63% top-1
and 0.68% top-5 accuracy improvement. In theory, SRFP and ASRFP are an
incremental regularization of the pruned filters. Besides, We note that SRFP
and ASRFP pursue better results while slowing down the speed of convergence.
