  Past years have witnessed rapid developments in Neural Machine Translation
(NMT). Most recently, with advanced modeling and training techniques, the
RNN-based NMT (RNMT) has shown its potential strength, even compared with the
well-known Transformer (self-attentional) model. Although the RNMT model can
possess very deep architectures through stacking layers, the transition depth
between consecutive hidden states along the sequential axis is still shallow.
In this paper, we further enhance the RNN-based NMT through increasing the
transition depth between consecutive hidden states and build a novel Deep
Transition RNN-based Architecture for Neural Machine Translation, named DTMT.
This model enhances the hidden-to-hidden transition with multiple non-linear
transformations, as well as maintains a linear transformation path throughout
this deep transition by the well-designed linear transformation mechanism to
alleviate the gradient vanishing problem. Experiments show that with the
specially designed deep transition modules, our DTMT can achieve remarkable
improvements on translation quality. Experimental results on Chinese->English
translation task show that DTMT can outperform the Transformer model by +2.09
BLEU points and achieve the best results ever reported in the same dataset. On
WMT14 English->German and English->French translation tasks, DTMT shows
superior quality to the state-of-the-art NMT systems, including the Transformer
and the RNMT+.
