  Script diversity presents a challenge to Multilingual Language Models (MLLM)
by reducing lexical overlap among closely related languages. Therefore,
transliterating closely related languages that use different writing scripts to
a common script may improve the downstream task performance of MLLMs. We
empirically measure the effect of transliteration on MLLMs in this context. We
specifically focus on the Indic languages, which have the highest script
diversity in the world, and we evaluate our models on the IndicGLUE benchmark.
We perform the Mann-Whitney U test to rigorously verify whether the effect of
transliteration is significant or not. We find that transliteration benefits
the low-resource languages without negatively affecting the comparatively
high-resource languages. We also measure the cross-lingual representation
similarity of the models using centered kernel alignment on parallel sentences
from the FLORES-101 dataset. We find that for parallel sentences across
different languages, the transliteration-based model learns sentence
representations that are more similar.
