  Knowledge distillation is commonly employed to compress neural networks,
reducing the inference costs and memory footprint. In the scenario of
homogenous architecture, feature-based methods have been widely validated for
their effectiveness. However, in scenarios where the teacher and student models
are of heterogeneous architectures, the inherent differences in feature
representation significantly degrade the performance of these methods. Recent
studies have highlighted that low-frequency components constitute the majority
of image features. Motivated by this, we propose a Low-Frequency
Components-based Contrastive Knowledge Distillation (LFCC) framework that
significantly enhances the performance of feature-based distillation between
heterogeneous architectures. Specifically, we designe a set of multi-scale
low-pass filters to extract the low-frequency components of intermediate
features from both the teacher and student models, aligning them in a compact
space to overcome architectural disparities. Moreover, leveraging the intrinsic
pairing characteristic of the teacher-student framework, we design an
innovative sample-level contrastive learning framework that adeptly
restructures the constraints of within-sample feature similarity and
between-sample feature divergence into a contrastive learning task. This
strategy enables the student model to capitalize on intra-sample feature
congruence while simultaneously enhancing the discrimination of features among
disparate samples. Consequently, our LFCC framework accurately captures the
commonalities in feature representation across heterogeneous architectures.
Extensive evaluations and empirical analyses across three architectures (CNNs,
Transformers, and MLPs) demonstrate that LFCC achieves superior performance on
the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be
publicly available.
