  While deep and large pre-trained models are the state-of-the-art for various
natural language processing tasks, their huge size poses significant challenges
for practical uses in resource constrained settings. Recent works in knowledge
distillation propose task-agnostic as well as task-specific methods to compress
these models, with task-specific ones often yielding higher compression rate.
In this work, we develop a new task-agnostic distillation framework
XtremeDistilTransformers that leverages the advantage of task-specific methods
for learning a small universal model that can be applied to arbitrary tasks and
languages. To this end, we study the transferability of several source tasks,
augmentation resources and model architecture for distillation. We evaluate our
model performance on multiple tasks, including the General Language
Understanding Evaluation (GLUE) benchmark, SQuAD question answering dataset and
a massive multi-lingual NER dataset with 41 languages. We release three
distilled task-agnostic checkpoints with 13MM, 22MM and 33MM parameters
obtaining SOTA performance in several tasks.
