  We study the effect of one type of imbalance often present in real-life
multilingual classification datasets: an uneven distribution of labels across
languages. We show evidence that fine-tuning a transformer-based Large Language
Model (LLM) on a dataset with this imbalance leads to worse performance, a more
pronounced separation of languages in the latent space, and the promotion of
uninformative features. We modify the traditional class weighing approach to
imbalance by calculating class weights separately for each language and show
that this helps mitigate those detrimental effects. These results create
awareness of the negative effects of language-specific class imbalance in
multilingual fine-tuning and the way in which the model learns to rely on the
separation of languages to perform the task.
