  We present a provable, sampling-based approach for generating compact
Convolutional Neural Networks (CNNs) by identifying and removing redundant
filters from an over-parameterized network. Our algorithm uses a small batch of
input data points to assign a saliency score to each filter and constructs an
importance sampling distribution where filters that highly affect the output
are sampled with correspondingly high probability. In contrast to existing
filter pruning approaches, our method is simultaneously data-informed, exhibits
provable guarantees on the size and performance of the pruned network, and is
widely applicable to varying network architectures and data sets. Our
analytical bounds bridge the notions of compressibility and importance of
network structures, which gives rise to a fully-automated procedure for
identifying and preserving filters in layers that are essential to the
network's performance. Our experimental evaluations on popular architectures
and data sets show that our algorithm consistently generates sparser and more
efficient models than those constructed by existing filter pruning approaches.
