  While prior work has established that the use of parallel data is conducive
for cross-lingual learning, it is unclear if the improvements come from the
data itself, or if it is the modeling of parallel interactions that matters.
Exploring this, we examine the usage of unsupervised machine translation to
generate synthetic parallel data, and compare it to supervised machine
translation and gold parallel data. We find that even model generated parallel
data can be useful for downstream tasks, in both a general setting (continued
pretraining) as well as the task-specific setting (translate-train), although
our best results are still obtained using real parallel data. Our findings
suggest that existing multilingual models do not exploit the full potential of
monolingual data, and prompt the community to reconsider the traditional
categorization of cross-lingual learning approaches.
