  Masked language models pick up gender biases during pre-training. Such biases
are usually attributed to a certain model architecture and its pre-training
corpora, with the implicit assumption that other variations in the pre-training
process, such as the choices of the random seed or the stopping point, have no
effect on the biases measured. However, we show that severe fluctuations exist
at the fundamental level of individual templates, invalidating the assumption.
Further against the intuition of how humans acquire biases, these fluctuations
are not correlated with the certainty of the predicted pronouns or the
profession frequencies in pre-training corpora. We release our code and data to
benefit future research.
