  Current medical artificial intelligence systems are often limited to narrow
applications, hindering their widespread adoption in clinical practice. To
address this limitation, we propose MedVersa, a generalist learner that enables
flexible learning and tasking for medical image interpretation. By leveraging a
large language model as a learnable orchestrator, MedVersa can learn from both
visual and linguistic supervision, support multimodal inputs, and perform
real-time task specification. This versatility allows MedVersa to adapt to
various clinical scenarios and perform multifaceted medical image analysis. We
introduce MedInterp, the largest multimodal dataset to date for medical image
interpretation, consisting of over 13 million annotated instances spanning 11
tasks across 3 modalities, to support the development of MedVersa. Our
experiments demonstrate that MedVersa achieves state-of-the-art performance in
9 tasks, sometimes outperforming specialist counterparts by over 10%. MedVersa
is the first to showcase the viability of multimodal generative medical AI in
implementing multimodal outputs, inputs, and dynamic task specification,
highlighting its potential as a multifunctional system for comprehensive
medical image analysis. This generalist approach to medical image
interpretation paves the way for more adaptable and efficient AI-assisted
clinical decision-making.
