  Multilingual neural machine translation models are trained to maximize the
likelihood of a mix of examples drawn from multiple language pairs. The
dominant inductive bias applied to these models is a shared vocabulary and a
shared set of parameters across languages; the inputs and labels corresponding
to examples drawn from different language pairs might still reside in distinct
sub-spaces. In this paper, we introduce multilingual crossover encoder-decoder
(mXEncDec) to fuse language pairs at an instance level. Our approach
interpolates instances from different language pairs into joint `crossover
examples' in order to encourage sharing input and output spaces across
languages. To ensure better fusion of examples in multilingual settings, we
propose several techniques to improve example interpolation across dissimilar
languages under heavy data imbalance. Experiments on a large-scale WMT
multilingual dataset demonstrate that our approach significantly improves
quality on English-to-Many, Many-to-English and zero-shot translation tasks
(from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets
demonstrate the capability of our approach to improve model generalization to
out-of-distribution multilingual examples. We also conduct qualitative and
quantitative representation comparisons to analyze the advantages of our
approach at the representation level.
