  We introduce a novel framework for incorporating human expertise into
algorithmic predictions. Our approach focuses on the use of human judgment to
distinguish inputs which `look the same' to any feasible predictive algorithm.
We argue that this framing clarifies the problem of human/AI collaboration in
prediction tasks, as experts often have access to information -- particularly
subjective information -- which is not encoded in the algorithm's training
data. We use this insight to develop a set of principled algorithms for
selectively incorporating human feedback only when it improves the performance
of any feasible predictor. We find empirically that although algorithms often
outperform their human counterparts on average, human judgment can
significantly improve algorithmic predictions on specific instances (which can
be identified ex-ante). In an X-ray classification task, we find that this
subset constitutes nearly 30% of the patient population. Our approach provides
a natural way of uncovering this heterogeneity and thus enabling effective
human-AI collaboration.
