  Multilingual pre-trained models are known to suffer from the curse of
multilinguality, which causes per-language performance to drop as they cover
more languages. We address this issue by introducing language-specific modules,
which allows us to grow the total capacity of the model, while keeping the
total number of trainable parameters per language constant. In contrast with
prior work that learns language-specific components post-hoc, we pre-train the
modules of our Cross-lingual Modular (X-Mod) models from the start. Our
experiments on natural language inference, named entity recognition and
question answering show that our approach not only mitigates the negative
interference between languages, but also enables positive transfer, resulting
in improved monolingual and cross-lingual performance. Furthermore, our
approach enables adding languages post-hoc with no measurable drop in
performance, no longer limiting the model usage to the set of pre-trained
languages.
