  When we rely on deep-learned models for robotic perception, we must recognize
that these models may behave unreliably on inputs dissimilar from the training
data, compromising the closed-loop system's safety. This raises fundamental
questions on how we can assess confidence in perception systems and to what
extent we can take safety-preserving actions when external environmental
changes degrade our perception model's performance. Therefore, we present a
framework to certify the safety of a perception-enabled system deployed in
novel contexts. To do so, we leverage robust model predictive control (MPC) to
control the system using the perception estimates while maintaining the
feasibility of a safety-preserving fallback plan that does not rely on the
perception system. In addition, we calibrate a runtime monitor using recently
proposed conformal prediction techniques to certifiably detect when the
perception system degrades beyond the tolerance of the MPC controller,
resulting in an end-to-end safety assurance. We show that this control
framework and calibration technique allows us to certify the system's safety
with orders of magnitudes fewer samples than required to retrain the perception
network when we deploy in a novel context on a photo-realistic aircraft taxiing
simulator. Furthermore, we illustrate the safety-preserving behavior of the MPC
on simulated examples of a quadrotor. We open-source our simulation platform
and provide videos of our results at our project page:
https://tinyurl.com/fallback-safe-mpc.
