  Recent studies have demonstrated the cross-lingual alignment ability of
multilingual pretrained language models. In this work, we found that the
cross-lingual alignment can be further improved by training seq2seq models on
sentence pairs mined using their own encoder outputs. We utilized these
findings to develop a new approach -- cross-lingual retrieval for iterative
self-supervised training (CRISS), where mining and training processes are
applied iteratively, improving cross-lingual alignment and translation ability
at the same time. Using this method, we achieved state-of-the-art unsupervised
machine translation results on 9 language directions with an average
improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the
XTREME benchmark on 16 languages with an average improvement of 21.5% in
absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU
improvement on average compared to mBART, when finetuned on supervised machine
translation downstream tasks.
