  Existing zero-shot cross-lingual transfer methods rely on parallel corpora or
bilingual dictionaries, which are expensive and impractical for low-resource
languages. To disengage from these dependencies, researchers have explored
training multilingual models on English-only resources and transferring them to
low-resource languages. However, its effect is limited by the gap between
embedding clusters of different languages. To address this issue, we propose
Embedding-Push, Attention-Pull, and Robust targets to transfer English
embeddings to virtual multilingual embeddings without semantic loss, thereby
improving cross-lingual transferability. Experimental results on mBERT and
XLM-R demonstrate that our method significantly outperforms previous works on
the zero-shot cross-lingual text classification task and can obtain a better
multilingual alignment.
