  The mainstream BERT/GPT model contains only 10 to 20 layers, and there is
little literature to discuss the training of deep BERT/GPT. This paper proposes
a simple yet effective method to stabilize BERT and GPT training. We
successfully scale up BERT and GPT to 1,000 layers, which is an order of
magnitude deeper than previous BERT and GPT. The proposed method
FoundationLayerNormalization enables efficient training of deep neural networks
and is validated at the 1000-layer scale.
