  Knowledge Distillation (KD) is extensively used to compress and deploy large
pre-trained language models on edge devices for real-world applications.
However, one neglected area of research is the impact of noisy (corrupted)
labels on KD. We present, to the best of our knowledge, the first study on KD
with noisy labels in Natural Language Understanding (NLU). We document the
scope of the problem and present two methods to mitigate the impact of label
noise. Experiments on the GLUE benchmark show that our methods are effective
even under high noise levels. Nevertheless, our results indicate that more
research is necessary to cope with label noise under the KD.
