  Pretrained language models are publicly available and constantly finetuned
for various real-life applications. As they become capable of grasping complex
contextual information, harmful biases are likely increasingly intertwined with
those models. This paper analyses gender bias in BERT models with two main
contributions: First, a novel bias measure is introduced, defining biases as
the difference in sentiment valuation of female and male sample versions.
Second, we comprehensively analyse BERT's biases on the example of a realistic
IMDB movie classifier. By systematically varying elements of the training
pipeline, we can conclude regarding their impact on the final model bias. Seven
different public BERT models in nine training conditions, i.e. 63 models in
total, are compared. Almost all conditions yield significant gender biases.
Results indicate that reflected biases stem from public BERT models rather than
task-specific data, emphasising the weight of responsible usage.
