  While search efficacy has been evaluated traditionally on the basis of result
relevance, fairness of search has attracted recent attention. In this work, we
define a notion of distributional fairness and provide a conceptual framework
for evaluating search results based on it. As part of this, we formulate a set
of axioms which an ideal evaluation framework should satisfy for distributional
fairness. We show how existing TREC test collections can be repurposed to study
fairness, and we measure potential data bias to inform test collection design
for fair search. A set of analyses show metric divergence between relevance and
fairness, and we describe a simple but flexible interpolation strategy for
integrating relevance and fairness into a single metric for optimization and
evaluation.
