  Effective human-AI collaboration requires a system design that provides
humans with meaningful ways to make sense of and critically evaluate
algorithmic recommendations. In this paper, we propose a way to augment
human-AI collaboration by building on a common organizational practice:
identifying experts who are likely to provide complementary opinions. When
machine learning algorithms are trained to predict human-generated assessments,
experts' rich multitude of perspectives is frequently lost in monolithic
algorithmic recommendations. The proposed approach aims to leverage productive
disagreement by (1) identifying whether some experts are likely to disagree
with an algorithmic assessment and, if so, (2) recommend an expert to request a
second opinion from.
