  The provided contents by information retrieval (IR) systems can reflect the
existing societal biases and stereotypes. Such biases in retrieval results can
lead to further establishing and strengthening stereotypes in society and also
in the systems. To facilitate the studies of gender bias in the retrieval
results of IR systems, we introduce Gender Representation-Bias for Information
Retrieval (Grep-BiasIR), a novel thoroughly-audited dataset consisting of 118
bias-sensitive neutral search queries. The set of queries covers a wide range
of gender-related topics, for which a biased representation of genders in the
search result can be considered as socially problematic. Each query is
accompanied with one relevant and one non-relevant document, where the document
is also provided in three variations of female, male, and neutral. The dataset
is available at https://github.com/KlaraKrieg/GrepBiasIR.
