  Vision Language Models (VLMs) like CLIP have attracted substantial attention
in pathology, serving as backbones for applications such as zero-shot image
classification and Whole Slide Image (WSI) analysis. Additionally, they can
function as vision encoders when combined with large language models (LLMs) to
support broader capabilities. Current efforts to train pathology VLMs rely on
pathology image-text pairs from platforms like PubMed, YouTube, and Twitter,
which provide limited, unscalable data with generally suboptimal image quality.
In this work, we leverage large-scale WSI datasets like TCGA to extract
numerous high-quality image patches. We then train a large multimodal model to
generate captions for these images, creating PathGen-1.6M, a dataset containing
1.6 million high-quality image-caption pairs. Our approach involves multiple
agent models collaborating to extract representative WSI patches, generating
and refining captions to obtain high-quality image-text pairs. Extensive
experiments show that integrating these generated pairs with existing datasets
to train a pathology-specific CLIP model, PathGen-CLIP, significantly enhances
its ability to analyze pathological images, with substantial improvements
across nine pathology-related zero-shot image classification tasks and three
whole-slide image tasks. Furthermore, we construct 200K instruction-tuning data
based on PathGen-1.6M and integrate PathGen-CLIP with the Vicuna LLM to create
more powerful multimodal models through instruction tuning. Overall, we provide
a scalable pathway for high-quality data generation in pathology, paving the
way for next-generation general pathology models.
