  Deep encoders have been proven to be effective in improving neural machine
translation (NMT) systems, but it reaches the upper bound of translation
quality when the number of encoder layers exceeds 18. Worse still, deeper
networks consume a lot of memory, making it impossible to train efficiently. In
this paper, we present Symbiosis Networks, which include a full network as the
Symbiosis Main Network (M-Net) and another shared sub-network with the same
structure but less layers as the Symbiotic Sub Network (S-Net). We adopt
Symbiosis Networks on Transformer-deep (m-n) architecture and define a
particular regularization loss $\mathcal{L}_{\tau}$ between the M-Net and S-Net
in NMT. We apply joint-training on the Symbiosis Networks and aim to improve
the M-Net performance. Our proposed training strategy improves Transformer-deep
(12-6) by 0.61, 0.49 and 0.69 BLEU over the baselines under classic training on
WMT'14 EN->DE, DE->EN and EN->FR tasks. Furthermore, our Transformer-deep
(12-6) even outperforms classic Transformer-deep (18-6).
