  The rapid advancement of large-scale vision-language models has showcased
remarkable capabilities across various tasks. However, the lack of extensive
and high-quality image-text data in medicine has greatly hindered the
development of large-scale medical vision-language models. In this work, we
present a diagnosis-guided bootstrapping strategy that exploits both image and
label information to construct vision-language datasets. Based on the
constructed dataset, we developed MedDr, a generalist foundation model for
healthcare capable of handling diverse medical data modalities, including
radiology, pathology, dermatology, retinography, and endoscopy. Moreover,
during inference, we propose a simple but effective retrieval-augmented medical
diagnosis strategy, which enhances the model's generalization ability.
Extensive experiments on visual question answering, medical report generation,
and medical image diagnosis demonstrate the superiority of our method.
