  The learning to defer (L2D) framework allows autonomous systems to be safe
and robust by allocating difficult decisions to a human expert. All existing
work on L2D assumes that each expert is well-identified, and if any expert were
to change, the system should be re-trained. In this work, we alleviate this
constraint, formulating an L2D system that can cope with never-before-seen
experts at test-time. We accomplish this by using meta-learning, considering
both optimization- and model-based variants. Given a small context set to
characterize the currently available expert, our framework can quickly adapt
its deferral policy. For the model-based approach, we employ an attention
mechanism that is able to look for points in the context set that are similar
to a given test point, leading to an even more precise assessment of the
expert's abilities. In the experiments, we validate our methods on image
recognition, traffic sign detection, and skin lesion diagnosis benchmarks.
