  Large-scale language models have recently demonstrated impressive empirical
performance. Nevertheless, the improved results are attained at the price of
bigger models, more power consumption, and slower inference, which hinder their
applicability to low-resource (both memory and computation) platforms.
Knowledge distillation (KD) has been demonstrated as an effective framework for
compressing such big models. However, large-scale neural network systems are
prone to memorize training instances, and thus tend to make inconsistent
predictions when the data distribution is altered slightly. Moreover, the
student model has few opportunities to request useful information from the
teacher model when there is limited task-specific data available. To address
these issues, we propose MixKD, a data-agnostic distillation framework that
leverages mixup, a simple yet efficient data augmentation approach, to endow
the resulting model with stronger generalization ability. Concretely, in
addition to the original training examples, the student model is encouraged to
mimic the teacher's behavior on the linear interpolation of example pairs as
well. We prove from a theoretical perspective that under reasonable conditions
MixKD gives rise to a smaller gap between the generalization error and the
empirical error. To verify its effectiveness, we conduct experiments on the
GLUE benchmark, where MixKD consistently leads to significant gains over the
standard KD training, and outperforms several competitive baselines.
Experiments under a limited-data setting and ablation studies further
demonstrate the advantages of the proposed approach.
