  Recent evidence reveals that Neural Machine Translation (NMT) models with
deeper neural networks can be more effective but are difficult to train. In
this paper, we present a MultiScale Collaborative (MSC) framework to ease the
training of NMT models that are substantially deeper than those used
previously. We explicitly boost the gradient back-propagation from top to
bottom levels by introducing a block-scale collaboration mechanism into deep
NMT models. Then, instead of forcing the whole encoder stack directly learns a
desired representation, we let each encoder block learns a fine-grained
representation and enhance it by encoding spatial dependencies using a
context-scale collaboration. We provide empirical evidence showing that the MSC
nets are easy to optimize and can obtain improvements of translation quality
from considerably increased depth. On IWSLT translation tasks with three
translation directions, our extremely deep models (with 72-layer encoders)
surpass strong baselines by +2.2~+3.1 BLEU points. In addition, our deep MSC
achieves a BLEU score of 30.56 on WMT14 English-German task that significantly
outperforms state-of-the-art deep NMT models.
