  A major focus of clinical imaging workflow is disease diagnosis and
management, leading to medical imaging datasets strongly tied to specific
clinical objectives. This scenario has led to the prevailing practice of
developing task-specific segmentation models, without gaining insights from
widespread imaging cohorts. Inspired by the training program of medical
radiology residents, we propose a shift towards universal medical image
segmentation, a paradigm aiming to build medical image understanding foundation
models by leveraging the diversity and commonality across clinical targets,
body regions, and imaging modalities. Towards this goal, we develop Hermes, a
novel context-prior learning approach to address the challenges of data
heterogeneity and annotation differences in medical image segmentation. In a
large collection of eleven diverse datasets (2,438 3D images) across five
modalities (CT, PET, T1, T2 and cine MRI) and multiple body regions, we
demonstrate the merit of the universal paradigm over the traditional paradigm
on addressing multiple tasks within a single model. By exploiting the synergy
across tasks, Hermes achieves state-of-the-art performance on all testing
datasets and shows superior model scalability. Results on two additional
datasets reveals Hermes' strong performance for transfer learning, incremental
learning, and generalization to downstream tasks. Hermes's learned priors
demonstrate an appealing trait to reflect the intricate relations among tasks
and modalities, which aligns with the established anatomical and imaging
principles in radiology. The code is available:
https://github.com/yhygao/universal-medical-image-segmentation.
