  Although large pre-trained language models have achieved great success in
many NLP tasks, it has been shown that they reflect human biases from their
pre-training corpora. This bias may lead to undesirable outcomes when these
models are applied in real-world settings. In this paper, we investigate the
bias present in monolingual BERT models across a diverse set of languages
(English, Greek, and Persian). While recent research has mostly focused on
gender-related biases, we analyze religious and ethnic biases as well and
propose a template-based method to measure any kind of bias, based on sentence
pseudo-likelihood, that can handle morphologically complex languages with
gender-based adjective declensions. We analyze each monolingual model via this
method and visualize cultural similarities and differences across different
dimensions of bias. Ultimately, we conclude that current methods of probing for
bias are highly language-dependent, necessitating cultural insights regarding
the unique ways bias is expressed in each language and culture (e.g. through
coded language, synecdoche, and other similar linguistic concepts). We also
hypothesize that higher measured social biases in the non-English BERT models
correlate with user-generated content in their training.
