  Multilingual generative models obtain remarkable cross-lingual in-context
learning capabilities through pre-training on large-scale corpora. However,
they still exhibit a performance bias toward high-resource languages and learn
isolated distributions of multilingual sentence representations, which may
hinder knowledge transfer across languages. To bridge this gap, we propose a
simple yet effective cross-lingual alignment framework exploiting pairs of
translation sentences. It aligns the internal sentence representations across
different languages via multilingual contrastive learning and aligns outputs by
following cross-lingual instructions in the target language. Experimental
results show that even with less than 0.1 {\textperthousand} of pre-training
tokens, our alignment framework significantly boosts the cross-lingual
abilities of generative language models and mitigates the performance gap.
Further analyses reveal that it results in a better internal multilingual
representation distribution of multilingual models.
