  Recent developments in machine translation and multilingual text generation
have led researchers to adopt trained metrics such as COMET or BLEURT, which
treat evaluation as a regression problem and use representations from
multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on
related tasks suggest that these models are most efficient when they are large,
which is costly and impractical for evaluation. We investigate the trade-off
between multilinguality and model capacity with RemBERT, a state-of-the-art
multilingual language model, using data from the WMT Metrics Shared Task. We
present a series of experiments which show that model size is indeed a
bottleneck for cross-lingual transfer, then demonstrate how distillation can
help addressing this bottleneck, by leveraging synthetic data generation and
transferring knowledge from one teacher to multiple students trained on related
languages. Our method yields up to 10.5% improvement over vanilla fine-tuning
and reaches 92.6% of RemBERT's performance using only a third of its
parameters.
