  When deploying machine learning models in high-stakes robotics applications,
the ability to detect unsafe situations is crucial. Early warning systems can
provide alerts when an unsafe situation is imminent (in the absence of
corrective action). To reliably improve safety, these warning systems should
have a provable false negative rate; i.e. of the situations that are unsafe,
fewer than $\epsilon$ will occur without an alert. In this work, we present a
framework that combines a statistical inference technique known as conformal
prediction with a simulator of robot/environment dynamics, in order to tune
warning systems to provably achieve an $\epsilon$ false negative rate using as
few as $1/\epsilon$ data points. We apply our framework to a driver warning
system and a robotic grasping application, and empirically demonstrate
guaranteed false negative rate while also observing low false detection
(positive) rate.
