  Massively multilingual Transformers (MMTs), such as mBERT and XLM-R, are
widely used for cross-lingual transfer learning. While these are pretrained to
represent hundreds of languages, end users of NLP systems are often interested
only in individual languages. For such purposes, the MMTs' language coverage
makes them unnecessarily expensive to deploy in terms of model size, inference
time, energy, and hardware cost. We thus propose to extract compressed,
language-specific models from MMTs which retain the capacity of the original
MMTs for cross-lingual transfer. This is achieved by distilling the MMT
bilingually, i.e., using data from only the source and target language of
interest. Specifically, we use a two-phase distillation approach, termed
BiStil: (i) the first phase distils a general bilingual model from the MMT,
while (ii) the second, task-specific phase sparsely fine-tunes the bilingual
"student" model using a task-tuned variant of the original MMT as its
"teacher". We evaluate this distillation technique in zero-shot cross-lingual
transfer across a number of standard cross-lingual benchmarks. The key results
indicate that the distilled models exhibit minimal degradation in target
language performance relative to the base MMT despite being significantly
smaller and faster. Furthermore, we find that they outperform multilingually
distilled models such as DistilmBERT and MiniLMv2 while having a very modest
training budget in comparison, even on a per-language basis. We also show that
bilingual models distilled from MMTs greatly outperform bilingual models
trained from scratch. Our code and models are available at
https://github.com/AlanAnsell/bistil.
