  Knowledge Distillation has shown very promising abil-ity in transferring
learned representation from the largermodel (teacher) to the smaller one
(student).Despitemany efforts, prior methods ignore the important role
ofretaining inter-channel correlation of features, leading tothe lack of
capturing intrinsic distribution of the featurespace and sufficient diversity
properties of features in theteacher network.To solve the issue, we propose
thenovel Inter-Channel Correlation for Knowledge Distillation(ICKD), with which
the diversity and homology of the fea-ture space of the student network can
align with that ofthe teacher network. The correlation between these
twochannels is interpreted as diversity if they are irrelevantto each other,
otherwise homology. Then the student isrequired to mimic the correlation within
its own embed-ding space. In addition, we introduce the grid-level
inter-channel correlation, making it capable of dense predictiontasks.
Extensive experiments on two vision tasks, includ-ing ImageNet classification
and Pascal VOC segmentation,demonstrate the superiority of our ICKD, which
consis-tently outperforms many existing methods, advancing thestate-of-the-art
in the fields of Knowledge Distillation. Toour knowledge, we are the first
method based on knowl-edge distillation boosts ResNet18 beyond 72% Top-1
ac-curacy on ImageNet classification. Code is available
at:https://github.com/ADLab-AutoDrive/ICKD.
