  Neural network pruning offers a promising prospect to facilitate deploying
deep neural networks on resource-limited devices. However, existing methods are
still challenged by the training inefficiency and labor cost in pruning
designs, due to missing theoretical guidance of non-salient network components.
In this paper, we propose a novel filter pruning method by exploring the High
Rank of feature maps (HRank). Our HRank is inspired by the discovery that the
average rank of multiple feature maps generated by a single filter is always
the same, regardless of the number of image batches CNNs receive. Based on
HRank, we develop a method that is mathematically formulated to prune filters
with low-rank feature maps. The principle behind our pruning is that low-rank
feature maps contain less information, and thus pruned results can be easily
reproduced. Besides, we experimentally show that weights with high-rank feature
maps contain more important information, such that even when a portion is not
updated, very little damage would be done to the model performance. Without
introducing any additional constraints, HRank leads to significant improvements
over the state-of-the-arts in terms of FLOPs and parameters reduction, with
similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs
reduction by removing 59.2% of the parameters, with only a small loss of 0.14%
in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction
by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1
accuracy on ImageNet. The codes can be available at
https://github.com/lmbxmu/HRank.
