  Sentiment analysis (SA) systems are used in many products and hundreds of
languages. Gender and racial biases are well-studied in English SA systems, but
understudied in other languages, with few resources for such studies. To remedy
this, we build a counterfactual evaluation corpus for gender and racial/migrant
bias in four languages. We demonstrate its usefulness by answering a simple but
important question that an engineer might need to answer when deploying a
system: What biases do systems import from pre-trained models when compared to
a baseline with no pre-training? Our evaluation corpus, by virtue of being
counterfactual, not only reveals which models have less bias, but also
pinpoints changes in model bias behaviour, which enables more targeted
mitigation strategies. We release our code and evaluation corpora to facilitate
future research.
