  Search Engines (SE) have been shown to perpetuate well-known gender
stereotypes identified in psychology literature and to influence users
accordingly. Similar biases were found encoded in Word Embeddings (WEs) learned
from large online corpora. In this context, we propose the Gender Stereotype
Reinforcement (GSR) measure, which quantifies the tendency of a SE to support
gender stereotypes, leveraging gender-related information encoded in WEs.
Through the critical lens of construct validity, we validate the proposed
measure on synthetic and real collections. Subsequently, we use GSR to compare
widely-used Information Retrieval ranking algorithms, including lexical,
semantic, and neural models. We check if and how ranking algorithms based on
WEs inherit the biases of the underlying embeddings. We also consider the most
common debiasing approaches for WEs proposed in the literature and test their
impact in terms of GSR and common performance measures. To the best of our
knowledge, GSR is the first specifically tailored measure for IR, capable of
quantifying representational harms.
