  Detection of Out-of-Distribution (OOD) samples in real time is a crucial
safety check for deployment of machine learning models in the medical field.
Despite a growing number of uncertainty quantification techniques, there is a
lack of evaluation guidelines on how to select OOD detection methods in
practice. This gap impedes implementation of OOD detection methods for
real-world applications. Here, we propose a series of practical considerations
and tests to choose the best OOD detector for a specific medical dataset. These
guidelines are illustrated on a real-life use case of Electronic Health Records
(EHR). Our results can serve as a guide for implementation of OOD detection
methods in clinical practice, mitigating risks associated with the use of
machine learning models in healthcare.
