  Large language models demonstrate reasonable multilingual abilities, despite
predominantly English-centric pretraining. However, the spontaneous
multilingual alignment in these models is shown to be weak, leading to
unsatisfactory cross-lingual transfer and knowledge sharing. Previous works
attempt to address this issue by explicitly injecting multilingual alignment
information during or after pretraining. Thus for the early stage in
pretraining, the alignment is weak for sharing information or knowledge across
languages. In this paper, we propose PreAlign, a framework that establishes
multilingual alignment prior to language model pretraining. PreAlign injects
multilingual alignment by initializing the model to generate similar
representations of aligned words and preserves this alignment using a
code-switching strategy during pretraining. Extensive experiments in a
synthetic English to English-Clone setting demonstrate that PreAlign
significantly outperforms standard multilingual joint training in language
modeling, zero-shot cross-lingual transfer, and cross-lingual knowledge
application. Further experiments in real-world scenarios further validate
PreAlign's effectiveness across various model sizes.
