  Knowledge distillation(KD) aims to improve the performance of a student
network by mimicing the knowledge from a powerful teacher network. Existing
methods focus on studying what knowledge should be transferred and treat all
samples equally during training. This paper introduces the adaptive sample
weighting to KD. We discover that previous effective hard mining methods are
not appropriate for distillation. Furthermore, we propose Prime-Aware Adaptive
Distillation (PAD) by the incorporation of uncertainty learning. PAD perceives
the prime samples in distillation and then emphasizes their effect adaptively.
PAD is fundamentally different from and would refine existing methods with the
innovative view of unequal training. For this reason, PAD is versatile and has
been applied in various tasks including classification, metric learning, and
object detection. With ten teacher-student combinations on six datasets, PAD
promotes the performance of existing distillation methods and outperforms
recent state-of-the-art methods.
