  Knowledge distillation addresses the problem of transferring knowledge from a
teacher model to a student model. In this process, we typically have multiple
types of knowledge extracted from the teacher model. The problem is to make
full use of them to train the student model. Our preliminary study shows that:
(1) not all of the knowledge is necessary for learning a good student model,
and (2) knowledge distillation can benefit from certain knowledge at different
training steps. In response to these, we propose an actor-critic approach to
selecting appropriate knowledge to transfer during the process of knowledge
distillation. In addition, we offer a refinement of the training algorithm to
ease the computational burden. Experimental results on the GLUE datasets show
that our method outperforms several strong knowledge distillation baselines
significantly.
