  In cross-lingual language models, representations for many different
languages live in the same space. Here, we investigate the linguistic and
non-linguistic factors affecting sentence-level alignment in cross-lingual
pretrained language models for 101 languages and 5,050 language pairs. Using
BERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our
corpus, we compute a task-based measure of cross-lingual alignment in the form
of bitext retrieval performance, as well as four intrinsic measures of vector
space alignment and isomorphism. We then examine a range of linguistic,
quasi-linguistic, and training-related features as potential predictors of
these alignment metrics. The results of our analyses show that word order
agreement and agreement in morphological complexity are two of the strongest
linguistic predictors of cross-linguality. We also note in-family training data
as a stronger predictor than language-specific training data across the board.
We verify some of our linguistic findings by looking at the effect of
morphological segmentation on English-Inuktitut alignment, in addition to
examining the effect of word order agreement on isomorphism for 66 zero-shot
language pairs from a different corpus. We make the data and code for our
experiments publicly available.
