  Loss spikes often occur during pre-training of large language models. The
spikes degrade the performance of large language models and sometimes ruin the
pre-training. Since the pre-training needs a vast computational budget, we
should avoid such spikes. To investigate the cause of loss spikes, we focus on
gradients of internal layers. Through theoretical analyses, we reveal two
causes of the exploding gradients, and provide requirements to prevent the
explosion. In addition, we propose a method to satisfy the requirements by
combining the initialization method and a simple modification to embeddings. We
conduct various experiments to verify our theoretical analyses empirically.
Experimental results indicate that the combination is effective in preventing
spikes during pre-training.
