  Does Knowledge Distillation (KD) really work? Conventional wisdom viewed it
as a knowledge transfer procedure where a perfect mimicry of the student to its
teacher is desired. However, paradoxical studies indicate that closely
replicating the teacher's behavior does not consistently improve student
generalization, posing questions on its possible causes. Confronted with this
gap, we hypothesize that diverse attentions in teachers contribute to better
student generalization at the expense of reduced fidelity in ensemble KD
setups. By increasing data augmentation strengths, our key findings reveal a
decrease in the Intersection over Union (IoU) of attentions between teacher
models, leading to reduced student overfitting and decreased fidelity. We
propose this low-fidelity phenomenon as an underlying characteristic rather
than a pathology when training KD. This suggests that stronger data
augmentation fosters a broader perspective provided by the divergent teacher
ensemble and lower student-teacher mutual information, benefiting
generalization performance. These insights clarify the mechanism on
low-fidelity phenomenon in KD. Thus, we offer new perspectives on optimizing
student model performance, by emphasizing increased diversity in teacher
attentions and reduced mimicry behavior between teachers and student.
