  In this work, we provide a large-scale empirical study of the scaling
properties of multilingual neural machine translation models. We examine how
increases in the model size affect the model performance and investigate the
role of the training mixture composition on the scaling behavior. We find that
changing the weightings of the individual language pairs in the training
mixture only affect the multiplicative factor of the scaling law. In
particular, we observe that multilingual models trained using different mixing
rates all exhibit the same scaling exponent. Through a novel joint scaling law
formulation, we compute the effective number of parameters allocated to each
language pair and examine the role of language similarity in the scaling
behavior of our models. We find little evidence that language similarity has
any impact. In contrast, the direction of the multilinguality plays a
significant role, with models translating from multiple languages into English
having a larger number of effective parameters per task than their reversed
counterparts. Finally, we leverage our observations to predict the performance
of multilingual models trained with any language weighting at any scale,
significantly reducing efforts required for language balancing in large
multilingual models. Our findings apply to both in-domain and out-of-domain
test sets and to multiple evaluation metrics, such as ChrF and BLEURT.
