  Learning-to-defer is a framework to automatically defer decision-making to a
human expert when ML-based decisions are deemed unreliable. Existing
learning-to-defer frameworks are not designed for sequential settings. That is,
they defer at every instance independently, based on immediate predictions,
while ignoring the potential long-term impact of these interventions. As a
result, existing frameworks are myopic. Further, they do not defer adaptively,
which is crucial when human interventions are costly. In this work, we propose
Sequential Learning-to-Defer (SLTD), a framework for learning-to-defer to a
domain expert in sequential decision-making settings. Contrary to existing
literature, we pose the problem of learning-to-defer as model-based
reinforcement learning (RL) to i) account for long-term consequences of
ML-based actions using RL and ii) adaptively defer based on the dynamics
(model-based). Our proposed framework determines whether to defer (at each time
step) by quantifying whether a deferral now will improve the value compared to
delaying deferral to the next time step. To quantify the improvement, we
account for potential future deferrals. As a result, we learn a pre-emptive
deferral policy (i.e. a policy that defers early if using the ML-based policy
could worsen long-term outcomes). Our deferral policy is adaptive to the
non-stationarity in the dynamics. We demonstrate that adaptive deferral via
SLTD provides an improved trade-off between long-term outcomes and deferral
frequency on synthetic, semi-synthetic, and real-world data with non-stationary
dynamics. Finally, we interpret the deferral decision by decomposing the
propagated (long-term) uncertainty around the outcome, to justify the deferral
decision.
