  As more researchers have become aware of and passionate about algorithmic
fairness, there has been an explosion in papers laying out new metrics,
suggesting algorithms to address issues, and calling attention to issues in
existing applications of machine learning. This research has greatly expanded
our understanding of the concerns and challenges in deploying machine learning,
but there has been much less work in seeing how the rubber meets the road.
  In this paper we provide a case-study on the application of fairness in
machine learning research to a production classification system, and offer new
insights in how to measure and address algorithmic fairness issues. We discuss
open questions in implementing equality of opportunity and describe our
fairness metric, conditional equality, that takes into account distributional
differences. Further, we provide a new approach to improve on the fairness
metric during model training and demonstrate its efficacy in improving
performance for a real-world product
