  Deep Neural Network (DNN) is powerful but computationally expensive and
memory intensive, thus impeding its practical usage on resource-constrained
front-end devices. DNN pruning is an approach for deep model compression, which
aims at eliminating some parameters with tolerable performance degradation. In
this paper, we propose a novel momentum-SGD-based optimization method to reduce
the network complexity by on-the-fly pruning. Concretely, given a global
compression ratio, we categorize all the parameters into two parts at each
training iteration which are updated using different rules. In this way, we
gradually zero out the redundant parameters, as we update them using only the
ordinary weight decay but no gradients derived from the objective function. As
a departure from prior methods that require heavy human works to tune the
layer-wise sparsity ratios, prune by solving complicated non-differentiable
problems or finetune the model after pruning, our method is characterized by 1)
global compression that automatically finds the appropriate per-layer sparsity
ratios; 2) end-to-end training; 3) no need for a time-consuming re-training
process after pruning; and 4) superior capability to find better winning
tickets which have won the initialization lottery.
