  The Transformer model has achieved state-of-the-art performance in many
sequence modeling tasks. However, how to leverage model capacity with large or
variable depths is still an open challenge. We present a probabilistic
framework to automatically learn which layer(s) to use by learning the
posterior distributions of layer selection. As an extension of this framework,
we propose a novel method to train one shared Transformer network for
multilingual machine translation with different layer selection posteriors for
each language pair. The proposed method alleviates the vanishing gradient issue
and enables stable training of deep Transformers (e.g. 100 layers). We evaluate
on WMT English-German machine translation and masked language modeling tasks,
where our method outperforms existing approaches for training deeper
Transformers. Experiments on multilingual machine translation demonstrate that
this approach can effectively leverage increased model capacity and bring
universal improvement for both many-to-one and one-to-many translation with
diverse language pairs.
