  BERT and other large-scale language models (LMs) contain gender and racial
bias. They also exhibit other dimensions of social bias, most of which have not
been studied in depth, and some of which vary depending on the language. In
this paper, we study ethnic bias and how it varies across languages by
analyzing and mitigating ethnic bias in monolingual BERT for English, German,
Spanish, Korean, Turkish, and Chinese. To observe and quantify ethnic bias, we
develop a novel metric called Categorical Bias score. Then we propose two
methods for mitigation; first using a multilingual model, and second using
contextual word alignment of two monolingual models. We compare our proposed
methods with monolingual BERT and show that these methods effectively alleviate
the ethnic bias. Which of the two methods works better depends on the amount of
NLP resources available for that language. We additionally experiment with
Arabic and Greek to verify that our proposed methods work for a wider variety
of languages.
