  Knowledge Distillation (KD) is a well-known training paradigm in deep neural
networks where knowledge acquired by a large teacher model is transferred to a
small student. KD has proven to be an effective technique to significantly
improve the student's performance for various tasks including object detection.
As such, KD techniques mostly rely on guidance at the intermediate feature
level, which is typically implemented by minimizing an lp-norm distance between
teacher and student activations during training. In this paper, we propose a
replacement for the pixel-wise independent lp-norm based on the structural
similarity (SSIM). By taking into account additional contrast and structural
cues, feature importance, correlation and spatial dependence in the feature
space are considered in the loss formulation. Extensive experiments on MSCOCO
demonstrate the effectiveness of our method across different training schemes
and architectures. Our method adds only little computational overhead, is
straightforward to implement and at the same time it significantly outperforms
the standard lp-norms. Moreover, more complex state-of-the-art KD methods using
attention-based sampling mechanisms are outperformed, including a +3.5 AP gain
using a Faster R-CNN R-50 compared to a vanilla model.
