  Language can be used as a means of reproducing and enforcing harmful
stereotypes and biases and has been analysed as such in numerous research. In
this paper, we present a survey of 304 papers on gender bias in natural
language processing. We analyse definitions of gender and its categories within
social sciences and connect them to formal definitions of gender bias in NLP
research. We survey lexica and datasets applied in research on gender bias and
then compare and contrast approaches to detecting and mitigating gender bias.
We find that research on gender bias suffers from four core limitations. 1)
Most research treats gender as a binary variable neglecting its fluidity and
continuity. 2) Most of the work has been conducted in monolingual setups for
English or other high-resource languages. 3) Despite a myriad of papers on
gender bias in NLP methods, we find that most of the newly developed algorithms
do not test their models for bias and disregard possible ethical considerations
of their work. 4) Finally, methodologies developed in this line of research are
fundamentally flawed covering very limited definitions of gender bias and
lacking evaluation baselines and pipelines. We suggest recommendations towards
overcoming these limitations as a guide for future research.
