  Knowledge distillation (KD) is a general neural network training approach
that uses a teacher model to guide the student model. Existing works mainly
study KD from the network output side (e.g., trying to design a better KD loss
function), while few have attempted to understand it from the input side.
Especially, its interplay with data augmentation (DA) has not been well
understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix)
inherently perform much better than others in KD? What makes a "good" DA in KD?
Our investigation from a statistical perspective suggests that a good DA scheme
should reduce the covariance of the teacher-student cross-entropy. A practical
metric, the stddev of teacher's mean probability (T. stddev), is further
presented and well justified empirically. Besides the theoretical
understanding, we also introduce a new entropy-based data-mixing DA scheme,
CutMixPick, to further enhance CutMix. Extensive empirical studies support our
claims and demonstrate how we can harvest considerable performance gains simply
by using a better DA scheme in knowledge distillation.
