  Knowledge Distillation (KD) has been extensively used for natural language
understanding (NLU) tasks to improve a small model's (a student) generalization
by transferring the knowledge from a larger model (a teacher). Although KD
methods achieve state-of-the-art performance in numerous settings, they suffer
from several problems limiting their performance. It is shown in the literature
that the capacity gap between the teacher and the student networks can make KD
ineffective. Additionally, existing KD techniques do not mitigate the noise in
the teacher's output: modeling the noisy behaviour of the teacher can distract
the student from learning more useful features. We propose a new KD method that
addresses these problems and facilitates the training compared to previous
techniques. Inspired by continuation optimization, we design a training
procedure that optimizes the highly non-convex KD objective by starting with
the smoothed version of this objective and making it more complex as the
training proceeds. Our method (Continuation-KD) achieves state-of-the-art
performance across various compact architectures on NLU (GLUE benchmark) and
computer vision tasks (CIFAR-10 and CIFAR-100).
