  Content-based image retrieval (CBIR) has the potential to significantly
improve diagnostic aid and medical research in radiology. Current CBIR systems
face limitations due to their specialization to certain pathologies, limiting
their utility. In response, we propose using vision foundation models as
powerful and versatile off-the-shelf feature extractors for content-based
medical image retrieval. By benchmarking these models on a comprehensive
dataset of 1.6 million 2D radiological images spanning four modalities and 161
pathologies, we identify weakly-supervised models as superior, achieving a P@1
of up to 0.594. This performance not only competes with a specialized model but
does so without the need for fine-tuning. Our analysis further explores the
challenges in retrieving pathological versus anatomical structures, indicating
that accurate retrieval of pathological features presents greater difficulty.
Despite these challenges, our research underscores the vast potential of
foundation models for CBIR in radiology, proposing a shift towards versatile,
general-purpose medical image retrieval systems that do not require specific
tuning.
