  Avoiding out-of-distribution (OOD) data is critical for training supervised
machine learning models in the medical imaging domain. Furthermore, obtaining
labeled medical data is difficult and expensive since it requires expert
annotators like doctors, radiologists, etc. Active learning (AL) is a
well-known method to mitigate labeling costs by selecting the most diverse or
uncertain samples. However, current AL methods do not work well in the medical
imaging domain with OOD data. We propose Diagnose (avoiDing out-of-dIstribution
dAta usinG submodular iNfOrmation meaSurEs), a novel active learning framework
that can jointly model similarity and dissimilarity, which is crucial in mining
in-distribution data and avoiding OOD data at the same time. Particularly, we
use a small number of data points as exemplars that represent a query set of
in-distribution data points and a private set of OOD data points. We illustrate
the generalizability of our framework by evaluating it on a wide variety of
real-world OOD scenarios. Our experiments verify the superiority of Diagnose
over the state-of-the-art AL methods across multiple domains of medical
imaging.
