  Masked Language Models (MLMs) have been successful in many natural language
processing tasks. However, real-world stereotype biases are likely to be
reflected in MLMs due to their learning from large text corpora. Most of the
evaluation metrics proposed in the past adopt different masking strategies,
designed with the log-likelihood of MLMs. They lack holistic considerations
such as variance for stereotype bias and anti-stereotype bias samples. In this
paper, the log-likelihoods of stereotype bias and anti-stereotype bias samples
output by MLMs are considered Gaussian distributions. Two evaluation metrics,
Kullback Leibler Divergence Score (KLDivS) and Jensen Shannon Divergence Score
(JSDivS) are proposed to evaluate social biases in MLMs The experimental
results on the public datasets StereoSet and CrowS-Pairs demonstrate that
KLDivS and JSDivS are more stable and interpretable compared to the metrics
proposed in the past.
