  Many technical approaches have been proposed for ensuring that decisions made
by machine learning systems are fair, but few of these proposals have been
stress-tested in real-world systems. This paper presents an example of one
team's approach to the challenge of applying algorithmic fairness approaches to
complex production systems within the context of a large technology company. We
discuss how we disentangle normative questions of product and policy design
(like, "how should the system trade off between different stakeholders'
interests and needs?") from empirical questions of system implementation (like,
"is the system achieving the desired tradeoff in practice?"). We also present
an approach for answering questions of the latter sort, which allows us to
measure how machine learning systems and human labelers are making these
tradeoffs across different relevant groups. We hope our experience integrating
fairness tools and approaches into large-scale and complex production systems
will be useful to other practitioners facing similar challenges, and
illuminating to academics and researchers looking to better address the needs
of practitioners.
