  Pre-trained multilingual language models have become an important building
block in multilingual natural language processing. In the present paper, we
investigate a range of such models to find out how well they transfer
discourse-level knowledge across languages. This is done with a systematic
evaluation on a broader set of discourse-level tasks than has been previously
been assembled. We find that the XLM-RoBERTa family of models consistently show
the best performance, by simultaneously being good monolingual models and
degrading relatively little in a zero-shot setting. Our results also indicate
that model distillation may hurt the ability of cross-lingual transfer of
sentence representations, while language dissimilarity at most has a modest
effect. We hope that our test suite, covering 5 tasks with a total of 22
languages in 10 distinct families, will serve as a useful evaluation platform
for multilingual performance at and beyond the sentence level.
