  English pretrained language models, which make up the backbone of many modern
NLP systems, require huge amounts of unlabeled training data. These models are
generally presented as being trained only on English text but have been found
to transfer surprisingly well to other languages. We investigate this
phenomenon and find that common English pretraining corpora actually contain
significant amounts of non-English text: even when less than 1% of data is not
English (well within the error rate of strong language classifiers), this leads
to hundreds of millions of foreign language tokens in large-scale datasets. We
then demonstrate that even these small percentages of non-English data
facilitate cross-lingual transfer for models trained on them, with target
language performance strongly correlated to the amount of in-language data seen
during pretraining. In light of these findings, we argue that no model is truly
monolingual when pretrained at scale, which should be considered when
evaluating cross-lingual transfer.
