  Robustness to out-of-distribution (OOD) samples is crucial for safely
deploying machine learning models in the open world. Recent works have focused
on designing scoring functions to quantify OOD uncertainty. Setting appropriate
thresholds for these scoring functions for OOD detection is challenging as OOD
samples are often unavailable up front. Typically, thresholds are set to
achieve a desired true positive rate (TPR), e.g., $95\%$ TPR. However, this can
lead to very high false positive rates (FPR), ranging from 60 to 96\%, as
observed in the Open-OOD benchmark. In safety-critical real-life applications,
e.g., medical diagnosis, controlling the FPR is essential when dealing with
various OOD samples dynamically. To address these challenges, we propose a
mathematically grounded OOD detection framework that leverages expert feedback
to \emph{safely} update the threshold on the fly. We provide theoretical
results showing that it is guaranteed to meet the FPR constraint at all times
while minimizing the use of human feedback. Another key feature of our
framework is that it can work with any scoring function for OOD uncertainty
quantification. Empirical evaluation of our system on synthetic and benchmark
OOD datasets shows that our method can maintain FPR at most $5\%$ while
maximizing TPR.
