  Over 85 million computed tomography (CT) scans are performed annually in the
US, of which approximately one quarter focus on the abdomen. Given the current
radiologist shortage, there is a large impetus to use artificial intelligence
to alleviate the burden of interpreting these complex imaging studies. Prior
state-of-the-art approaches for automated medical image interpretation leverage
vision language models (VLMs). However, current medical VLMs are generally
limited to 2D images and short reports, and do not leverage electronic health
record (EHR) data for supervision. We introduce Merlin - a 3D VLM that we train
using paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes
(1.8+ million codes), and radiology reports (6+ million tokens). We evaluate
Merlin on 6 task types and 752 individual tasks. The non-adapted
(off-the-shelf) tasks include zero-shot findings classification (31 findings),
phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval
(image to findings and image to impressions), while model adapted tasks include
5-year disease prediction (6 diseases), radiology report generation, and 3D
semantic segmentation (20 organs). We perform internal validation on a test set
of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public
CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant
evaluations, we assess the efficacy of various network architectures and
training strategies to depict that Merlin has favorable performance to existing
task-specific baselines. We derive data scaling laws to empirically assess
training data needs for requisite downstream task performance. Furthermore,
unlike conventional VLMs that require hundreds of GPUs for training, we perform
all training on a single GPU.
