  The dominant neural machine translation models are based on the
encoder-decoder structure, and many of them rely on an unconstrained receptive
field over source and target sequences. In this paper we study a new
architecture that breaks with both conventions. Our simplified architecture
consists in the decoder part of a transformer model, based on self-attention,
but with locality constraints applied on the attention receptive field. As
input for training, both source and target sentences are fed to the network,
which is trained as a language model. At inference time, the target tokens are
predicted autoregressively starting with the source sequence as previous
tokens. The proposed model achieves a new state of the art of 35.7 BLEU on
IWSLT'14 German-English and matches the best reported results in the literature
on the WMT'14 English-German and WMT'14 English-French translation benchmarks.
