  Pre-trained multilingual language encoders, such as multilingual BERT and
XLM-R, show great potential for zero-shot cross-lingual transfer. However,
these multilingual encoders do not precisely align words and phrases across
languages. Especially, learning alignments in the multilingual embedding space
usually requires sentence-level or word-level parallel corpora, which are
expensive to be obtained for low-resource languages. An alternative is to make
the multilingual encoders more robust; when fine-tuning the encoder using
downstream task, we train the encoder to tolerate noise in the contextual
embedding spaces such that even if the representations of different languages
are not aligned well, the model can still achieve good performance on zero-shot
cross-lingual transfer. In this work, we propose a learning strategy for
training robust models by drawing connections between adversarial examples and
the failure cases of zero-shot cross-lingual transfer. We adopt two widely used
robust training methods, adversarial training and randomized smoothing, to
train the desired robust model. The experimental results demonstrate that
robust training improves zero-shot cross-lingual transfer on text
classification tasks. The improvement is more significant in the generalized
cross-lingual transfer setting, where the pair of input sentences belong to two
different languages.
