  We propose a two-stage approach for training a single NMT model to translate
unseen languages both to and from English. For the first stage, we initialize
an encoder-decoder model to pretrained XLM-R and RoBERTa weights, then perform
multilingual fine-tuning on parallel data in 40 languages to English. We find
this model can generalize to zero-shot translations on unseen languages. For
the second stage, we leverage this generalization ability to generate synthetic
parallel data from monolingual datasets, then bidirectionally train with
successive rounds of back-translation.
  Our approach, which we EcXTra (English-centric Crosslingual (X) Transfer), is
conceptually simple, only using a standard cross-entropy objective throughout.
It is also data-driven, sequentially leveraging auxiliary parallel data and
monolingual data. We evaluate unsupervised NMT results for 7 low-resource
languages, and find that each round of back-translation training further
refines bidirectional performance. Our final single EcXTra-trained model
achieves competitive translation performance in all translation directions,
notably establishing a new state-of-the-art for English-to-Kazakh (22.9 > 10.4
BLEU). Our code is available at https://github.com/manestay/EcXTra .
