  Online platforms mediate access to opportunity: relevance-based rankings
create and constrain options by allocating exposure to job openings and job
candidates in hiring platforms, or sellers in a marketplace. In order to do so
responsibly, these socially consequential systems employ various fairness
measures and interventions, many of which seek to allocate exposure based on
worthiness. Because these constructs are typically not directly observable,
platforms must instead resort to using proxy scores such as relevance and infer
them from behavioral signals such as searcher clicks. Yet, it remains an open
question whether relevance fulfills its role as such a worthiness score in
high-stakes fair rankings. In this paper, we combine perspectives and tools
from the social sciences, information retrieval, and fairness in machine
learning to derive a set of desired criteria that relevance scores should
satisfy in order to meaningfully guide fairness interventions. We then
empirically show that not all of these criteria are met in a case study of
relevance inferred from biased user click data. We assess the impact of these
violations on the estimated system fairness and analyze whether existing
fairness interventions may mitigate the identified issues. Our analyses and
results surface the pressing need for new approaches to relevance collection
and generation that are suitable for use in fair ranking.
