  Multilingual Neural Machine Translation (MNMT) enables one system to
translate sentences from multiple source languages to multiple target
languages, greatly reducing deployment costs compared with conventional
bilingual systems. The MNMT training benefit, however, is often limited to
many-to-one directions. The model suffers from poor performance in one-to-many
and many-to-many with zero-shot setup. To address this issue, this paper
discusses how to practically build MNMT systems that serve arbitrary X-Y
translation directions while leveraging multilinguality with a two-stage
training strategy of pretraining and finetuning. Experimenting with the WMT'21
multilingual translation task, we demonstrate that our systems outperform the
conventional baselines of direct bilingual models and pivot translation models
for most directions, averagely giving +6.0 and +4.1 BLEU, without the need for
architecture change or extra data collection. Moreover, we also examine our
proposed approach in an extremely large-scale data setting to accommodate
practical deployment scenarios.
