  Recent studies have demonstrated how to assess the stereotypical bias in
pre-trained English language models. In this work, we extend this branch of
research in multiple different dimensions by systematically investigating (a)
mono- and multilingual models of (b) different underlying architectures with
respect to their bias in (c) multiple different languages. To that end, we make
use of the English StereoSet data set (Nadeem et al., 2021), which we
semi-automatically translate into German, French, Spanish, and Turkish. We find
that it is of major importance to conduct this type of analysis in a
multilingual setting, as our experiments show a much more nuanced picture as
well as notable differences from the English-only analysis. The main takeaways
from our analysis are that mGPT-2 (partly) shows surprising anti-stereotypical
behavior across languages, English (monolingual) models exhibit the strongest
bias, and the stereotypes reflected in the data set are least present in
Turkish models. Finally, we release our codebase alongside the translated data
sets and practical guidelines for the semi-automatic translation to encourage a
further extension of our work to other languages.
