  Recent years have witnessed the rapid advance in neural machine translation
(NMT), the core of which lies in the encoder-decoder architecture. Inspired by
the recent progress of large-scale pre-trained language models on machine
translation in a limited scenario, we firstly demonstrate that a single
language model (LM4MT) can achieve comparable performance with strong
encoder-decoder NMT models on standard machine translation benchmarks, using
the same training data and similar amount of model parameters. LM4MT can also
easily utilize source-side texts as additional supervision. Though modeling the
source- and target-language texts with the same mechanism, LM4MT can provide
unified representations for both source and target sentences, which can better
transfer knowledge across languages. Extensive experiments on pivot-based and
zero-shot translation tasks show that LM4MT can outperform the encoder-decoder
NMT model by a large margin.
