  Oversight AI is an emerging concept in radiology where the AI forms a
symbiosis with radiologists by continuously supporting radiologists in their
decision-making. Recent advances in vision-language models sheds a light on the
long-standing problems of the oversight AI by the understanding both visual and
textual concepts and their semantic correspondences. However, there have been
limited successes in the application of vision-language models in the medical
domain, as the current vision-language models and learning strategies for
photographic images and captions call for the web-scale data corpus of image
and text pairs which was not often feasible in the medical domain. To address
this, here we present a model dubbed Medical Cross-attention Vision-Language
model (Medical X-VL), leveraging the key components to be tailored for the
medical domain. Our medical X-VL model is based on the following components:
self-supervised uni-modal models in medical domain and fusion encoder to bridge
them, momentum distillation, sentence-wise contrastive learning for medical
reports, and the sentence similarity-adjusted hard negative mining. We
experimentally demonstrated that our model enables various zero-shot tasks for
oversight AI, ranging from the zero-shot classification to zero-shot error
correction. Our model outperformed the current state-of-the-art models in two
different medical image database, suggesting the novel clinical usage of our
oversight AI model for monitoring human errors. Our method was especially
successful in the data-limited setting, which is frequently encountered in the
clinics, suggesting the potential widespread applicability in medical domain.
