  Knowledge Distillation (KD) aims at improving the performance of a
low-capacity student model by inheriting knowledge from a high-capacity teacher
model. Previous KD methods typically train a student by minimizing a
task-related loss and the KD loss simultaneously, using a pre-defined loss
weight to balance these two terms. In this work, we propose to first transfer
the backbone knowledge from a teacher to the student, and then only learn the
task-head of the student network. Such a decomposition of the training process
circumvents the need of choosing an appropriate loss weight, which is often
difficult in practice, and thus makes it easier to apply to different datasets
and tasks. Importantly, the decomposition permits the core of our method,
Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive
feature mimicking from teacher to student. Extensive experiments on CIFAR-100
and ImageNet suggest that SSKD significantly narrows down the performance gap
between student and teacher, outperforming state-of-the-art approaches. We also
demonstrate the generalization ability of SSKD on other challenging benchmarks,
including face recognition on IJB-A dataset as well as object detection on COCO
dataset.
