  This work investigates the effect of gender-stereotypical biases in the
content of retrieved results on the relevance judgement of users/annotators. In
particular, since relevance in information retrieval (IR) is a
multi-dimensional concept, we study whether the value and quality of the
retrieved documents for some bias-sensitive queries can be judged differently
when the content of the documents represents different genders. To this aim, we
conduct a set of experiments where the genders of the participants are known as
well as experiments where the participants genders are not specified. The set
of experiments comprise of retrieval tasks, where participants perform a rated
relevance judgement for different search query and search result document
compilations. The shown documents contain different gender indications and are
either relevant or non-relevant to the query. The results show the differences
between the average judged relevance scores among documents with various gender
contents. Our work initiates further research on the connection of the
perception of gender stereotypes in users with their judgements and effects on
IR systems, and aim to raise awareness about the possible biases in this
domain.
