  Robot learning methods have recently made great strides, but generalization
and robustness challenges still hinder their widespread deployment. Failing to
detect and address potential failures renders state-of-the-art learning systems
not combat-ready for high-stakes tasks. Recent advances in interactive
imitation learning have presented a promising framework for human-robot
teaming, enabling the robots to operate safely and continually improve their
performances over long-term deployments. Nonetheless, existing methods
typically require constant human supervision and preemptive feedback, limiting
their practicality in realistic domains. This work aims to endow a robot with
the ability to monitor and detect errors during task execution. We introduce a
model-based runtime monitoring algorithm that learns from deployment data to
detect system anomalies and anticipate failures. Unlike prior work that cannot
foresee future failures or requires failure experiences for training, our
method learns a latent-space dynamics model and a failure classifier, enabling
our method to simulate future action outcomes and detect out-of-distribution
and high-risk states preemptively. We train our method within an interactive
imitation learning framework, where it continually updates the model from the
experiences of the human-robot team collected using trustworthy deployments.
Consequently, our method reduces the human workload needed over time while
ensuring reliable task execution. Our method outperforms the baselines across
system-level and unit-test metrics, with 23% and 40% higher success rates in
simulation and on physical hardware, respectively. More information at
https://ut-austin-rpl.github.io/sirius-runtime-monitor/
