  Knowledge distillation is a popular technique for training a small student
network to emulate a larger teacher model, such as an ensemble of networks. We
show that while knowledge distillation can improve student generalization, it
does not typically work as it is commonly understood: there often remains a
surprisingly large discrepancy between the predictive distributions of the
teacher and the student, even in cases when the student has the capacity to
perfectly match the teacher. We identify difficulties in optimization as a key
reason for why the student is unable to match the teacher. We also show how the
details of the dataset used for distillation play a role in how closely the
student matches the teacher -- and that more closely matching the teacher
paradoxically does not always lead to better student generalization.
