  Large multilingual pretrained language models (mPLMs) have become the de
facto state of the art for cross-lingual transfer in NLP. However, their
large-scale deployment to many languages, besides pretraining data scarcity, is
also hindered by the increase in vocabulary size and limitations in their
parameter budget. In order to boost the capacity of mPLMs to deal with
low-resource and unseen languages, we explore the potential of leveraging
transliteration on a massive scale. In particular, we explore the UROMAN
transliteration tool, which provides mappings from UTF-8 to Latin characters
for all the writing systems, enabling inexpensive romanization for virtually
any language. We first focus on establishing how UROMAN compares against other
language-specific and manually curated transliterators for adapting
multilingual PLMs. We then study and compare a plethora of data- and
parameter-efficient strategies for adapting the mPLMs to romanized and
non-romanized corpora of 14 diverse low-resource languages. Our results reveal
that UROMAN-based transliteration can offer strong performance for many
languages, with particular gains achieved in the most challenging setups: on
languages with unseen scripts and with limited training data without any
vocabulary augmentation. Further analyses reveal that an improved tokenizer
based on romanized data can even outperform non-transliteration-based methods
in the majority of languages.
