  Cross-lingual transfer learning is an important property of multilingual
large language models (LLMs). But how do LLMs represent relationships between
languages? Every language model has an input layer that maps tokens to vectors.
This ubiquitous layer of language models is often overlooked. We find that
similarities between these input embeddings are highly interpretable and that
the geometry of these embeddings differs between model families. In one case
(XLM-RoBERTa), embeddings encode language: tokens in different writing systems
can be linearly separated with an average of 99.2% accuracy. Another family
(mT5) represents cross-lingual semantic similarity: the 50 nearest neighbors
for any token represent an average of 7.61 writing systems, and are frequently
translations. This result is surprising given that there is no explicit
parallel cross-lingual training corpora and no explicit incentive for
translations in pre-training objectives. Our research opens the door for
investigations in 1) The effect of pre-training and model architectures on
representations of languages and 2) The applications of cross-lingual
representations embedded in language models.
