  We introduce a novel yet straightforward neural network initialization scheme
that modifies conventional methods like Xavier and Kaiming initialization.
Inspired by the concept of emergence and leveraging the emergence measures
proposed by Li (2023), our method adjusts the layer-wise weight scaling factors
to achieve higher emergence values. This enhancement is easy to implement,
requiring no additional optimization steps for initialization compared to
GradInit. We evaluate our approach across various architectures, including MLP
and convolutional architectures for image recognition, and transformers for
machine translation. We demonstrate substantial improvements in both model
accuracy and training speed, with and without batch normalization. The
simplicity, theoretical innovation, and demonstrable empirical advantages of
our method make it a potent enhancement to neural network initialization
practices. These results suggest a promising direction for leveraging emergence
to improve neural network training methodologies. Code is available at:
https://github.com/johnnyjingzeli/EmergenceInit.
