  In the light of the fact that the stochastic gradient descent (SGD) often
finds a flat minimum valley in the training loss, we propose a novel
directional pruning method which searches for a sparse minimizer in or close to
that flat region. The proposed pruning method does not require retraining or
the expert knowledge on the sparsity level. To overcome the computational
formidability of estimating the flat directions, we propose to use a carefully
tuned $\ell_1$ proximal gradient algorithm which can provably achieve the
directional pruning with a small learning rate after sufficient training. The
empirical results demonstrate the promising results of our solution in highly
sparse regime (92% sparsity) among many existing pruning methods on the
ResNet50 with the ImageNet, while using only a slightly higher wall time and
memory footprint than the SGD. Using the VGG16 and the wide ResNet 28x10 on the
CIFAR-10 and CIFAR-100, we demonstrate that our solution reaches the same
minima valley as the SGD, and the minima found by our solution and the SGD do
not deviate in directions that impact the training loss. The code that
reproduces the results of this paper is available at
https://github.com/donlan2710/gRDA-Optimizer/tree/master/directional_pruning.
