  The amount of medical images for training deep classification models is
typically very scarce, making these deep models prone to overfit the training
data. Studies showed that knowledge distillation (KD), especially the
mean-teacher framework which is more robust to perturbations, can help mitigate
the over-fitting effect. However, directly transferring KD from computer vision
to medical image classification yields inferior performance as medical images
suffer from higher intra-class variance and class imbalance. To address these
issues, we propose a novel Categorical Relation-preserving Contrastive
Knowledge Distillation (CRCKD) algorithm, which takes the commonly used
mean-teacher model as the supervisor. Specifically, we propose a novel
Class-guided Contrastive Distillation (CCD) module to pull closer positive
image pairs from the same class in the teacher and student models, while
pushing apart negative image pairs from different classes. With this
regularization, the feature distribution of the student model shows higher
intra-class similarity and inter-class variance. Besides, we propose a
Categorical Relation Preserving (CRP) loss to distill the teacher's relational
knowledge in a robust and class-balanced manner. With the contribution of the
CCD and CRP, our CRCKD algorithm can distill the relational knowledge more
comprehensively. Extensive experiments on the HAM10000 and APTOS datasets
demonstrate the superiority of the proposed CRCKD method.
