  Recently, it was shown that the role of the teacher in knowledge distillation
(KD) is to provide the student with an estimate of the true Bayes conditional
probability density (BCPD). Notably, the new findings propose that the
student's error rate can be upper-bounded by the mean squared error (MSE)
between the teacher's output and BCPD. Consequently, to enhance KD efficacy,
the teacher should be trained such that its output is close to BCPD in MSE
sense. This paper elucidates that training the teacher model with MSE loss
equates to minimizing the MSE between its output and BCPD, aligning with its
core responsibility of providing the student with a BCPD estimate closely
resembling it in MSE terms. In this respect, through a comprehensive set of
experiments, we demonstrate that substituting the conventional teacher trained
with cross-entropy loss with one trained using MSE loss in state-of-the-art KD
methods consistently boosts the student's accuracy, resulting in improvements
of up to 2.6\%.
