  Knowledge distillation (KD) has become an important technique for model
compression and knowledge transfer. In this work, we first perform a
comprehensive analysis of the knowledge transferred by different KD methods. We
demonstrate that traditional KD methods, which minimize the KL divergence of
softmax outputs between networks, are related to the knowledge alignment of an
individual sample only. Meanwhile, recent contrastive learning-based KD methods
mainly transfer relational knowledge between different samples, namely,
knowledge correlation. While it is important to transfer the full knowledge
from teacher to student, we introduce the Multi-level Knowledge Distillation
(MLKD) by effectively considering both knowledge alignment and correlation.
MLKD is task-agnostic and model-agnostic, and can easily transfer knowledge
from supervised or self-supervised pretrained teachers. We show that MLKD can
improve the reliability and transferability of learned representations.
Experiments demonstrate that MLKD outperforms other state-of-the-art methods on
a large number of experimental settings including different (a) pretraining
strategies (b) network architectures (c) datasets (d) tasks.
