  Knowledge distillation (KD) is an efficient framework for compressing
large-scale pre-trained language models. Recent years have seen a surge of
research aiming to improve KD by leveraging Contrastive Learning, Intermediate
Layer Distillation, Data Augmentation, and Adversarial Training. In this work,
we propose a learning based data augmentation technique tailored for knowledge
distillation, called CILDA. To the best of our knowledge, this is the first
time that intermediate layer representations of the main task are used in
improving the quality of augmented samples. More precisely, we introduce an
augmentation technique for KD based on intermediate layer matching using
contrastive loss to improve masked adversarial data augmentation. CILDA
outperforms existing state-of-the-art KD approaches on the GLUE benchmark, as
well as in an out-of-domain evaluation.
