  There are two primary approaches to addressing cross-lingual transfer:
multilingual pre-training, which implicitly aligns the hidden representations
of various languages, and translate-test, which explicitly translates different
languages into an intermediate language, such as English. Translate-test offers
better interpretability compared to multilingual pre-training. However, it has
lower performance than multilingual pre-training(Conneau and Lample, 2019;
Conneau et al, 2020) and struggles with word-level tasks due to translation
altering word order. As a result, we propose a new Machine-created Universal
Language (MUL) as an alternative intermediate language. MUL comprises a set of
discrete symbols forming a universal vocabulary and a natural language to MUL
translator for converting multiple natural languages to MUL. MUL unifies shared
concepts from various languages into a single universal word, enhancing
cross-language transfer. Additionally, MUL retains language-specific words and
word order, allowing the model to be easily applied to word-level tasks. Our
experiments demonstrate that translating into MUL yields improved performance
compared to multilingual pre-training, and our analysis indicates that MUL
possesses strong interpretability. The code is at:
https://github.com/microsoft/Unicoder/tree/master/MCUL.
