  Adversarial Robustness Distillation (ARD) is a novel method to boost the
robustness of small models. Unlike general adversarial training, its robust
knowledge transfer can be less easily restricted by the model capacity.
However, the teacher model that provides the robustness of knowledge does not
always make correct predictions, interfering with the student's robust
performances. Besides, in the previous ARD methods, the robustness comes
entirely from one-to-one imitation, ignoring the relationship between examples.
To this end, we propose a novel structured ARD method called Contrastive
Relationship DeNoise Distillation (CRDND). We design an adaptive compensation
module to model the instability of the teacher. Moreover, we utilize the
contrastive relationship to explore implicit robustness knowledge among
multiple examples. Experimental results on multiple attack benchmarks show
CRDND can transfer robust knowledge efficiently and achieves state-of-the-art
performances.
