  Structured pruning, especially channel pruning is widely used for the reduced
computational cost and the compatibility with off-the-shelf hardware devices.
Among existing works, weights are typically removed using a predefined global
threshold, or a threshold computed from a predefined metric. The predefined
global threshold based designs ignore the variation among different layers and
weights distribution, therefore, they may often result in sub-optimal
performance caused by over-pruning or under-pruning. In this paper, we present
a simple yet effective method, termed Optimal Thresholding (OT), to prune
channels with layer dependent thresholds that optimally separate important from
negligible channels. By using OT, most negligible or unimportant channels are
pruned to achieve high sparsity while minimizing performance degradation. Since
most important weights are preserved, the pruned model can be further
fine-tuned and quickly converge with very few iterations. Our method
demonstrates superior performance, especially when compared to the
state-of-the-art designs at high levels of sparsity. On CIFAR-100, a pruned and
fine-tuned DenseNet-121 by using OT achieves 75.99% accuracy with only 1.46e8
FLOPs and 0.71M parameters.
