  The "massively-multilingual" training of multilingual models is known to
limit their utility in any one language, and they perform particularly poorly
on low-resource languages. However, there is evidence that low-resource
languages can benefit from targeted multilinguality, where the model is trained
on closely related languages. To test this approach more rigorously, we
systematically study best practices for adapting a pre-trained model to a
language family. Focusing on the Uralic family as a test case, we adapt XLM-R
under various configurations to model 15 languages; we then evaluate the
performance of each experimental setting on two downstream tasks and 11
evaluation languages. Our adapted models significantly outperform mono- and
multilingual baselines. Furthermore, a regression analysis of hyperparameter
effects reveals that adapted vocabulary size is relatively unimportant for
low-resource languages, and that low-resource languages can be aggressively
up-sampled during training at little detriment to performance in high-resource
languages. These results introduce new best practices for performing language
adaptation in a targeted setting.
