  In recent years, deep learning has made remarkable progress in a wide range
of domains, with a particularly notable impact on natural language processing
tasks. One of the challenges associated with training deep neural networks in
the context of LLMs is the need for large amounts of computational resources
and time. To mitigate this, network growing algorithms offer potential cost
savings, but their underlying mechanisms are poorly understood. We present two
notable contributions in this paper. First, we present Deep Fusion, an
efficient approach to network training that leverages pre-trained
initializations of smaller networks. Second, we propose a theoretical framework
using backward error analysis to illustrate the dynamics of mid-training
network growth. Our experiments show how Deep Fusion is a practical and
effective approach that not only accelerates the training process but also
reduces computational requirements, maintaining or surpassing traditional
training methods' performance in various NLP tasks and T5 model sizes. Finally,
we validate our theoretical framework, which guides the optimal use of Deep
Fusion, showing that with carefully optimized training dynamics, it
significantly reduces both training time and resource consumption.
