  Machine learning (ML) models are increasingly being used in application
domains that often involve working together with human experts. In this
context, it can be advantageous to defer certain instances to a single human
expert when they are difficult to predict for the ML model. While previous work
has focused on scenarios with one distinct human expert, in many real-world
situations several human experts with varying capabilities may be available. In
this work, we propose an approach that trains a classification model to
complement the capabilities of multiple human experts. By jointly training the
classifier together with an allocation system, the classifier learns to
accurately predict those instances that are difficult for the human experts,
while the allocation system learns to pass each instance to the most suitable
team member -- either the classifier or one of the human experts. We evaluate
our proposed approach in multiple experiments on public datasets with
"synthetic" experts and a real-world medical dataset annotated by multiple
radiologists. Our approach outperforms prior work and is more accurate than the
best human expert or a classifier. Furthermore, it is flexibly adaptable to
teams of varying sizes and different levels of expert diversity.
