  Knowledge distillation aims to compress a powerful yet cumbersome teacher
model into a lightweight student model without much sacrifice of performance.
For this purpose, various approaches have been proposed over the past few
years, generally with elaborately designed knowledge representations, which in
turn increase the difficulty of model development and interpretation. In
contrast, we empirically show that a simple knowledge distillation technique is
enough to significantly narrow down the teacher-student performance gap. We
directly reuse the discriminative classifier from the pre-trained teacher model
for student inference and train a student encoder through feature alignment
with a single $\ell_2$ loss. In this way, the student model is able to achieve
exactly the same performance as the teacher model provided that their extracted
features are perfectly aligned. An additional projector is developed to help
the student encoder match with the teacher classifier, which renders our
technique applicable to various teacher and student architectures. Extensive
experiments demonstrate that our technique achieves state-of-the-art results at
the modest cost of compression ratio due to the added projector.
