  Knowledge distillation often involves how to define and transfer knowledge
from teacher to student effectively. Although recent self-supervised
contrastive knowledge achieves the best performance, forcing the network to
learn such knowledge may damage the representation learning of the original
class recognition task. We therefore adopt an alternative self-supervised
augmented task to guide the network to learn the joint distribution of the
original recognition task and self-supervised auxiliary task. It is
demonstrated as a richer knowledge to improve the representation power without
losing the normal classification capability. Moreover, it is incomplete that
previous methods only transfer the probabilistic knowledge between the final
layers. We propose to append several auxiliary classifiers to hierarchical
intermediate feature maps to generate diverse self-supervised knowledge and
perform the one-to-one transfer to teach the student network thoroughly. Our
method significantly surpasses the previous SOTA SSKD with an average
improvement of 2.56\% on CIFAR-100 and an improvement of 0.77\% on ImageNet
across widely used network pairs. Codes are available at
https://github.com/winycg/HSAKD.
