  Soft threshold pruning is among the cutting-edge pruning methods with
state-of-the-art performance. However, previous methods either perform aimless
searching on the threshold scheduler or simply set the threshold trainable,
lacking theoretical explanation from a unified perspective. In this work, we
reformulate soft threshold pruning as an implicit optimization problem solved
using the Iterative Shrinkage-Thresholding Algorithm (ISTA), a classic method
from the fields of sparse recovery and compressed sensing. Under this
theoretical framework, all threshold tuning strategies proposed in previous
studies of soft threshold pruning are concluded as different styles of tuning
$L_1$-regularization term. We further derive an optimal threshold scheduler
through an in-depth study of threshold scheduling based on our framework. This
scheduler keeps $L_1$-regularization coefficient stable, implying a
time-invariant objective function from the perspective of optimization. In
principle, the derived pruning algorithm could sparsify any mathematical model
trained via SGD. We conduct extensive experiments and verify its
state-of-the-art performance on both Artificial Neural Networks (ResNet-50 and
MobileNet-V1) and Spiking Neural Networks (SEW ResNet-18) on ImageNet datasets.
On the basis of this framework, we derive a family of pruning methods,
including sparsify-during-training, early pruning, and pruning at
initialization. The code is available at https://github.com/Yanqi-Chen/LATS.
