  Transfer learning between different language pairs has shown its
effectiveness for Neural Machine Translation (NMT) in low-resource scenario.
However, existing transfer methods involving a common target language are far
from success in the extreme scenario of zero-shot translation, due to the
language space mismatch problem between transferor (the parent model) and
transferee (the child model) on the source side. To address this challenge, we
propose an effective transfer learning approach based on cross-lingual
pre-training. Our key idea is to make all source languages share the same
feature space and thus enable a smooth transition for zero-shot translation. To
this end, we introduce one monolingual pre-training method and two bilingual
pre-training methods to obtain a universal encoder for different languages.
Once the universal encoder is constructed, the parent model built on such
encoder is trained with large-scale annotated data and then directly applied in
zero-shot translation scenario. Experiments on two public datasets show that
our approach significantly outperforms strong pivot-based baseline and various
multilingual NMT approaches.
