  Overparameterized transformer networks have obtained state of the art results
in various natural language processing tasks, such as machine translation,
language modeling, and question answering. These models contain hundreds of
millions of parameters, necessitating a large amount of computation and making
them prone to overfitting. In this work, we explore LayerDrop, a form of
structured dropout, which has a regularization effect during training and
allows for efficient pruning at inference time. In particular, we show that it
is possible to select sub-networks of any depth from one large network without
having to finetune them and with limited impact on performance. We demonstrate
the effectiveness of our approach by improving the state of the art on machine
translation, language modeling, summarization, question answering, and language
understanding benchmarks. Moreover, we show that our approach leads to small
BERT-like models of higher quality compared to training from scratch or using
distillation.
