  Filter pruning simultaneously accelerates the computation and reduces the
memory overhead of CNNs, which can be effectively applied to edge devices and
cloud services. In this paper, we propose a novel Knowledge-driven Differential
Filter Sampler~(KDFS) with Masked Filter Modeling~(MFM) framework for filter
pruning, which globally prunes the redundant filters based on the prior
knowledge of a pre-trained model in a differential and non-alternative
optimization. Specifically, we design a differential sampler with learnable
sampling parameters to build a binary mask vector for each layer, determining
whether the corresponding filters are redundant. To learn the mask, we
introduce masked filter modeling to construct PCA-like knowledge by aligning
the intermediate features from the pre-trained teacher model and the outputs of
the student decoder taking sampling features as the input. The mask and sampler
are directly optimized by the Gumbel-Softmax Straight-Through Gradient
Estimator in an end-to-end manner in combination with global pruning
constraint, MFM reconstruction error, and dark knowledge. Extensive experiments
demonstrate the proposed KDFS's effectiveness in compressing the base models on
various datasets. For instance, the pruned ResNet-50 on ImageNet achieves
$55.36\%$ computation reduction, and $42.86\%$ parameter reduction, while only
dropping $0.35\%$ Top-1 accuracy, significantly outperforming the
state-of-the-art methods. The code is available at
\url{https://github.com/Osilly/KDFS}.
