  Content Warning: This paper contains examples of misgendering and erasure
that could be offensive and potentially triggering.
  Gender bias in language technologies has been widely studied, but research
has mostly been restricted to a binary paradigm of gender. It is essential also
to consider non-binary gender identities, as excluding them can cause further
harm to an already marginalized group. In this paper, we comprehensively
evaluate popular language models for their ability to correctly use English
gender-neutral pronouns (e.g., singular they, them) and neo-pronouns (e.g., ze,
xe, thon) that are used by individuals whose gender identity is not represented
by binary pronouns. We introduce MISGENDERED, a framework for evaluating large
language models' ability to correctly use preferred pronouns, consisting of (i)
instances declaring an individual's pronoun, followed by a sentence with a
missing pronoun, and (ii) an experimental setup for evaluating masked and
auto-regressive language models using a unified method. When prompted
out-of-the-box, language models perform poorly at correctly predicting
neo-pronouns (averaging 7.7% accuracy) and gender-neutral pronouns (averaging
34.2% accuracy). This inability to generalize results from a lack of
representation of non-binary pronouns in training data and memorized
associations. Few-shot adaptation with explicit examples in the prompt improves
performance for neo-pronouns, but only to 64.7% even with 20 shots. We release
the full dataset, code, and demo at
https://tamannahossainkay.github.io/misgendered/
