  Cross-lingual transfer learning without labeled target language data or
parallel text has been surprisingly effective in zero-shot cross-lingual
classification, question answering, unsupervised machine translation, etc.
However, some recent publications have claimed that domain mismatch prevents
cross-lingual transfer, and their results show that unsupervised bilingual
lexicon induction (UBLI) and unsupervised neural machine translation (UNMT) do
not work well when the underlying monolingual corpora come from different
domains (e.g., French text from Wikipedia but English text from UN
proceedings). In this work, we show that a simple initialization regimen can
overcome much of the effect of domain mismatch in cross-lingual transfer. We
pre-train word and contextual embeddings on the concatenated domain-mismatched
corpora, and use these as initializations for three tasks: MUSE UBLI, UN
Parallel UNMT, and the SemEval 2017 cross-lingual word similarity task. In all
cases, our results challenge the conclusions of prior work by showing that
proper initialization can recover a large portion of the losses incurred by
domain mismatch.
