  Sense embedding learning methods learn different embeddings for the different
senses of an ambiguous word. One sense of an ambiguous word might be socially
biased while its other senses remain unbiased. In comparison to the numerous
prior work evaluating the social biases in pretrained word embeddings, the
biases in sense embeddings have been relatively understudied. We create a
benchmark dataset for evaluating the social biases in sense embeddings and
propose novel sense-specific bias evaluation measures. We conduct an extensive
evaluation of multiple static and contextualised sense embeddings for various
types of social biases using the proposed measures. Our experimental results
show that even in cases where no biases are found at word-level, there still
exist worrying levels of social biases at sense-level, which are often ignored
by the word-level bias evaluation measures.
