  Convolutional neural networks (CNNs) are typically over-parameterized,
bringing considerable computational overhead and memory footprint in inference.
Pruning a proportion of unimportant filters is an efficient way to mitigate the
inference cost. For this purpose, identifying unimportant convolutional filters
is the key to effective filter pruning. Previous work prunes filters according
to either their weight norms or the corresponding batch-norm scaling factors,
while neglecting the sequential dependency between adjacent layers. In this
paper, we further develop the norm-based importance estimation by taking the
dependency between the adjacent layers into consideration. Besides, we propose
a novel mechanism to dynamically control the sparsity-inducing regularization
so as to achieve the desired sparsity. In this way, we can identify unimportant
filters and search for the optimal network architecture within certain resource
budgets in a more principled manner. Comprehensive experimental results
demonstrate the proposed method performs favorably against the existing strong
baseline on the CIFAR, SVHN, and ImageNet datasets. The training sources will
be publicly available after the review process.
