  Filter level pruning is an effective method to accelerate the inference speed
of deep CNN models. Although numerous pruning algorithms have been proposed,
there are still two open issues. The first problem is how to prune residual
connections. We propose to prune both channels inside and outside the residual
connections via a KL-divergence based criterion. The second issue is pruning
with limited data. We observe an interesting phenomenon: directly pruning on a
small dataset is usually worse than fine-tuning a small model which is pruned
or trained from scratch on the large dataset. Knowledge distillation is an
effective approach to compensate for the weakness of limited data. However, the
logits of a teacher model may be noisy. In order to avoid the influence of
label noise, we propose a label refinement approach to solve this problem.
Experiments have demonstrated the effectiveness of our method (CURL,
Compression Using Residual-connections and Limited-data). CURL significantly
outperforms previous state-of-the-art methods on ImageNet. More importantly,
when pruning on small datasets, CURL achieves comparable or much better
performance than fine-tuning a pretrained small model.
