  Stacking non-linear layers allows deep neural networks to model complicated
functions, and including residual connections in Transformer layers is
beneficial for convergence and performance. However, residual connections may
make the model "forget" distant layers and fail to fuse information from
previous layers effectively. Selectively managing the representation
aggregation of Transformer layers may lead to better performance. In this
paper, we present a Transformer with depth-wise LSTMs connecting cascading
Transformer layers and sub-layers. We show that layer normalization and
feed-forward computation within a Transformer layer can be absorbed into
depth-wise LSTMs connecting pure Transformer attention layers. Our experiments
with the 6-layer Transformer show significant BLEU improvements in both WMT 14
English-German / French tasks and the OPUS-100 many-to-many multilingual NMT
task, and our deep Transformer experiments demonstrate the effectiveness of
depth-wise LSTM on the convergence and performance of deep Transformers.
