  Many measures of societal bias in language models have been proposed in
recent years. A popular approach is to use a set of word filling prompts to
evaluate the behavior of the language models. In this work, we analyze the
validity of two such measures -- StereoSet and CrowS-Pairs. We show that these
measures produce unexpected and illogical results when appropriate control
group samples are constructed. Based on this, we believe that they are
problematic and using them in the future should be reconsidered. We propose a
way forward with an improved testing protocol. Finally, we also introduce a new
gender bias dataset for Slovak.
