  Multilingual pre-trained models exhibit zero-shot cross-lingual transfer,
where a model fine-tuned on a source language achieves surprisingly good
performance on a target language. While studies have attempted to understand
transfer, they focus only on MLM, and the large number of differences between
natural languages makes it hard to disentangle the importance of different
properties. In this work, we specifically highlight the importance of word
embedding alignment by proposing a pre-training objective (ALIGN-MLM) whose
auxiliary loss guides similar words in different languages to have similar word
embeddings. ALIGN-MLM either outperforms or matches three widely adopted
objectives (MLM, XLM, DICT-MLM) when we evaluate transfer between pairs of
natural languages and their counterparts created by systematically modifying
specific properties like the script. In particular, ALIGN-MLM outperforms XLM
and MLM by 35 and 30 F1 points on POS-tagging for transfer between languages
that differ both in their script and word order (left-to-right v.s.
right-to-left). We also show a strong correlation between alignment and
transfer for all objectives (e.g., rho=0.727 for XNLI), which together with
ALIGN-MLM's strong performance calls for explicitly aligning word embeddings
for multilingual models.
