  Policy distillation in deep reinforcement learning provides an effective way
to transfer control policies from a larger network to a smaller untrained
network without a significant degradation in performance. However, policy
distillation is underexplored in deep reinforcement learning, and existing
approaches are computationally inefficient, resulting in a long distillation
time. In addition, the effectiveness of the distillation process is still
limited to the model capacity. We propose a new distillation mechanism, called
real-time policy distillation, in which training the teacher model and
distilling the policy to the student model occur simultaneously. Accordingly,
the teacher's latest policy is transferred to the student model in real time.
This reduces the distillation time to half the original time or even less and
also makes it possible for extremely small student models to learn skills at
the expert level. We evaluated the proposed algorithm in the Atari 2600 domain.
The results show that our approach can achieve full distillation in most games,
even with compression ratios up to 1.7%.
