  Learning algorithms are often used in conjunction with expert decision makers
in practical scenarios, however this fact is largely ignored when designing
these algorithms. In this paper we explore how to learn predictors that can
either predict or choose to defer the decision to a downstream expert. Given
only samples of the expert's decisions, we give a procedure based on learning a
classifier and a rejector and analyze it theoretically. Our approach is based
on a novel reduction to cost sensitive learning where we give a consistent
surrogate loss for cost sensitive learning that generalizes the cross entropy
loss. We show the effectiveness of our approach on a variety of experimental
tasks.
