  Pre-trained language models have been known to perpetuate biases from the
underlying datasets to downstream tasks. However, these findings are
predominantly based on monolingual language models for English, whereas there
are few investigative studies of biases encoded in language models for
languages beyond English. In this paper, we fill this gap by analysing gender
bias in West Slavic language models. We introduce the first template-based
dataset in Czech, Polish, and Slovak for measuring gender bias towards male,
female and non-binary subjects. We complete the sentences using both mono- and
multilingual language models and assess their suitability for the masked
language modelling objective. Next, we measure gender bias encoded in West
Slavic language models by quantifying the toxicity and genderness of the
generated words. We find that these language models produce hurtful completions
that depend on the subject's gender. Perhaps surprisingly, Czech, Slovak, and
Polish language models produce more hurtful completions with men as subjects,
which, upon inspection, we find is due to completions being related to
violence, death, and sickness.
