  Sparsification is an efficient approach to accelerate CNN inference, but it
is challenging to take advantage of sparsity in training procedure because the
involved gradients are dynamically changed. Actually, an important observation
shows that most of the activation gradients in back-propagation are very close
to zero and only have a tiny impact on weight-updating. Hence, we consider
pruning these very small gradients randomly to accelerate CNN training
according to the statistical distribution of activation gradients. Meanwhile,
we theoretically analyze the impact of pruning algorithm on the convergence.
The proposed approach is evaluated on AlexNet and ResNet-\{18, 34, 50, 101,
152\} with CIFAR-\{10, 100\} and ImageNet datasets. Experimental results show
that our training approach could substantially achieve up to $5.92 \times$
speedups at back-propagation stage with negligible accuracy loss.
