  Pretrained multilingual models have become a de facto default approach for
zero-shot cross-lingual transfer. Previous work has shown that these models are
able to achieve cross-lingual representations when pretrained on two or more
languages with shared parameters. In this work, we provide evidence that a
model can achieve language-agnostic representations even when pretrained on a
single language. That is, we find that monolingual models pretrained and
finetuned on different languages achieve competitive performance compared to
the ones that use the same target language. Surprisingly, the models show a
similar performance on a same task regardless of the pretraining language. For
example, models pretrained on distant languages such as German and Portuguese
perform similarly on English tasks.
