  Knowledge Distillation (KD) is a model-agnostic technique to improve model
quality while having a fixed capacity budget. It is a commonly used technique
for model compression, where a larger capacity teacher model with better
quality is used to train a more compact student model with better inference
efficiency. Through distillation, one hopes to benefit from student's
compactness, without sacrificing too much on model quality. Despite the large
success of knowledge distillation, better understanding of how it benefits
student model's training dynamics remains under-explored. In this paper, we
categorize teacher's knowledge into three hierarchical levels and study its
effects on knowledge distillation: (1) knowledge of the `universe', where KD
brings a regularization effect through label smoothing; (2) domain knowledge,
where teacher injects class relationships prior to student's logit layer
geometry; and (3) instance specific knowledge, where teacher rescales student
model's per-instance gradients based on its measurement on the event
difficulty. Using systematic analyses and extensive empirical studies on both
synthetic and real-world datasets, we confirm that the aforementioned three
factors play a major role in knowledge distillation. Furthermore, based on our
findings, we diagnose some of the failure cases of applying KD from recent
studies.
