  In recent years, recommendation and ranking systems have become increasingly
popular on digital platforms. However, previous work has highlighted how
personalized systems might lead to unintentional harms for users. Practitioners
require metrics to measure and mitigate these types of harms in production
systems. To meet this need, many fairness definitions have been introduced and
explored by the RecSys community. Unfortunately, this has led to a
proliferation of possible fairness metrics from which practitioners can choose.
The increase in volume and complexity of metrics creates a need for
practitioners to deeply understand the nuances of fairness definitions and
implementations. Additionally, practitioners need to understand the ethical
guidelines that accompany these metrics for responsible implementation. Recent
work has shown that there is a proliferation of ethics guidelines and has
pointed to the need for more implementation guidance rather than principles
alone. The wide variety of available metrics, coupled with the lack of accepted
standards or shared knowledge in practice leads to a challenging environment
for practitioners to navigate. In this position paper, we focus on this
widening gap between the research community and practitioners concerning the
availability of metrics versus the ability to put them into practice. We
address this gap with our current work, which focuses on developing methods to
help ML practitioners in their decision-making processes when picking fairness
metrics for recommendation and ranking systems. In our iterative design
interviews, we have already found that practitioners need both practical and
reflective guidance when refining fairness constraints. This is especially
salient given the growing challenge for practitioners to leverage the correct
metrics while balancing complex fairness contexts.
