  Knowledge distillation has been shown to be a powerful model compression
approach to facilitate the deployment of pre-trained language models in
practice. This paper focuses on task-agnostic distillation. It produces a
compact pre-trained model that can be easily fine-tuned on various tasks with
small computational costs and memory footprints. Despite the practical
benefits, task-agnostic distillation is challenging. Since the teacher model
has a significantly larger capacity and stronger representation power than the
student model, it is very difficult for the student to produce predictions that
match the teacher's over a massive amount of open-domain training data. Such a
large prediction discrepancy often diminishes the benefits of knowledge
distillation. To address this challenge, we propose Homotopic Distillation
(HomoDistil), a novel task-agnostic distillation approach equipped with
iterative pruning. Specifically, we initialize the student model from the
teacher model, and iteratively prune the student's neurons until the target
width is reached. Such an approach maintains a small discrepancy between the
teacher's and student's predictions throughout the distillation process, which
ensures the effectiveness of knowledge transfer. Extensive experiments
demonstrate that HomoDistil achieves significant improvements on existing
baselines.
