  With the emerging needs of creating fairness-aware solutions for search and
recommendation systems, a daunting challenge exists of evaluating such
solutions. While many of the traditional information retrieval (IR) metrics can
capture the relevance, diversity, and novelty for the utility with respect to
users, they are not suitable for inferring whether the presented results are
fair from the perspective of responsible information exposure. On the other
hand, existing fairness metrics do not account for user utility or do not
measure it adequately. To address this problem, we propose a new metric called
FAIR. By unifying standard IR metrics and fairness measures into an integrated
metric, this metric offers a new perspective for evaluating fairness-aware
ranking results. Based on this metric, we developed an effective ranking
algorithm that jointly optimized user utility and fairness. The experimental
results showed that our FAIR metric could highlight results with good user
utility and fair information exposure. We showed how FAIR related to a set of
existing utility and fairness metrics and demonstrated the effectiveness of our
FAIR-based algorithm. We believe our work opens up a new direction of pursuing
a metric for evaluating and implementing the FAIR systems.
