  Recent research suggests that combining AI models with a human expert can
exceed the performance of either alone. The combination of their capabilities
is often realized by learning to defer algorithms that enable the AI to learn
to decide whether to make a prediction for a particular instance or defer it to
the human expert. However, to accurately learn which instances should be
deferred to the human expert, a large number of expert predictions that
accurately reflect the expert's capabilities are required -- in addition to the
ground truth labels needed to train the AI. This requirement shared by many
learning to defer algorithms hinders their adoption in scenarios where the
responsible expert regularly changes or where acquiring a sufficient number of
expert predictions is costly. In this paper, we propose a three-step approach
to reduce the number of expert predictions required to train learning to defer
algorithms. It encompasses (1) the training of an embedding model with ground
truth labels to generate feature representations that serve as a basis for (2)
the training of an expertise predictor model to approximate the expert's
capabilities. (3) The expertise predictor generates artificial expert
predictions for instances not yet labeled by the expert, which are required by
the learning to defer algorithms. We evaluate our approach on two public
datasets. One with "synthetically" generated human experts and another from the
medical domain containing real-world radiologists' predictions. Our experiments
show that the approach allows the training of various learning to defer
algorithms with a minimal number of human expert predictions. Furthermore, we
demonstrate that even a small number of expert predictions per class is
sufficient for these algorithms to exceed the performance the AI and the human
expert can achieve individually.
