  Unpredictable ML model behavior on unseen data, especially in the health
domain, raises serious concerns about its safety as repercussions for mistakes
can be fatal. In this paper, we explore the feasibility of using
state-of-the-art out-of-distribution detectors for reliable and trustworthy
diagnostic predictions. We select publicly available deep learning models
relating to various health conditions (e.g., skin cancer, lung sound, and
Parkinson's disease) using various input data types (e.g., image, audio, and
motion data). We demonstrate that these models show unreasonable predictions on
out-of-distribution datasets. We show that Mahalanobis distance- and Gram
matrices-based out-of-distribution detection methods are able to detect
out-of-distribution data with high accuracy for the health models that operate
on different modalities. We then translate the out-of-distribution score into a
human interpretable CONFIDENCE SCORE to investigate its effect on the users'
interaction with health ML applications. Our user study shows that the
\textsc{confidence score} helped the participants only trust the results with a
high score to make a medical decision and disregard results with a low score.
Through this work, we demonstrate that dataset shift is a critical piece of
information for high-stake ML applications, such as medical diagnosis and
healthcare, to provide reliable and trustworthy predictions to the users.
