  In this study, we aim to build up a model that can Segment Anything in
radiology scans, driven by Text prompts, termed as SAT. Our main contributions
are three folds: (i) for dataset construction, we construct the first
multi-modal knowledge tree on human anatomy, including 6502 anatomical
terminologies; Then we build up the largest and most comprehensive segmentation
dataset for training, by collecting over 22K 3D medical image scans from 72
segmentation datasets, across 497 classes, with careful standardization on both
image scans and label space; (ii) for architecture design, we propose to inject
medical knowledge into a text encoder via contrastive learning, and then
formulate a universal segmentation model, that can be prompted by feeding in
medical terminologies in text form; (iii) As a result, we have trained SAT-Nano
(110M parameters) and SAT-Pro (447M parameters), demonstrating comparable
performance to 72 specialist nnU-Nets trained on each dataset/subsets. We
validate SAT as a foundational segmentation model, with better generalization
ability on external (unseen) datasets, and can be further improved on specific
tasks after fine-tuning adaptation. Comparing with interactive segmentation
model, for example, MedSAM, segmentation model prompted by text enables
superior performance, scalability and robustness. As a use case, we demonstrate
that SAT can act as a powerful out-of-the-box agent for large language models,
enabling visual grounding in clinical procedures such as report generation. All
the data, codes, and models in this work have been released.
