  Gating modules have been widely explored in dynamic network pruning to reduce
the run-time computational cost of deep neural networks while preserving the
representation of features. Despite the substantial progress, existing methods
remain ignoring the consistency between feature and gate distributions, which
may lead to distortion of gated features. In this paper, we propose a
feature-gate coupling (FGC) approach aiming to align distributions of features
and gates. FGC is a plug-and-play module, which consists of two steps carried
out in an iterative self-supervised manner. In the first step, FGC utilizes the
$k$-Nearest Neighbor method in the feature space to explore instance
neighborhood relationships, which are treated as self-supervisory signals. In
the second step, FGC exploits contrastive learning to regularize gating modules
with generated self-supervisory signals, leading to the alignment of instance
neighborhood relationships within the feature and gate spaces. Experimental
results validate that the proposed FGC method improves the baseline approach
with significant margins, outperforming the state-of-the-arts with better
accuracy-computation trade-off. Code is publicly available.
