  Optimizing a given metric is a central aspect of most current AI approaches,
yet overemphasizing metrics leads to manipulation, gaming, a myopic focus on
short-term goals, and other unexpected negative consequences. This poses a
fundamental contradiction for AI development. Through a series of real-world
case studies, we look at various aspects of where metrics go wrong in practice
and aspects of how our online environment and current business practices are
exacerbating these failures. Finally, we propose a framework towards mitigating
the harms caused by overemphasis of metrics within AI by: (1) using a slate of
metrics to get a fuller and more nuanced picture, (2) combining metrics with
qualitative accounts, and (3) involving a range of stakeholders, including
those who will be most impacted.
