  In a wide array of areas, algorithms are matching and surpassing the
performance of human experts, leading to consideration of the roles of human
judgment and algorithmic prediction in these domains. The discussion around
these developments, however, has implicitly equated the specific task of
prediction with the general task of automation. We argue here that automation
is broader than just a comparison of human versus algorithmic performance on a
task; it also involves the decision of which instances of the task to give to
the algorithm in the first place. We develop a general framework that poses
this latter decision as an optimization problem, and we show how basic
heuristics for this optimization problem can lead to performance gains even on
heavily-studied applications of AI in medicine. Our framework also serves to
highlight how effective automation depends crucially on estimating both
algorithmic and human error on an instance-by-instance basis, and our results
show how improvements in these error estimation problems can yield significant
gains for automation as well.
