  We propose a simple and efficient approach for training the BERT model. Our
approach exploits the special structure of BERT that contains a stack of
repeated modules (i.e., transformer encoders). Our proposed approach first
trains BERT with the weights shared across all the repeated modules till some
point. This is for learning the commonly shared component of weights across all
repeated layers. We then stop weight sharing and continue training until
convergence. We present theoretic insights for training by sharing weights then
unsharing with analysis for simplified models. Empirical experiments on the
BERT model show that our method yields better performance of trained models,
and significantly reduces the number of training iterations.
