  As advances in large language models (LLMs) and multimodal techniques
continue to mature, the development of general-purpose multimodal large
language models (MLLMs) has surged, offering significant applications in
interpreting natural images. However, the field of pathology has largely
remained untapped, particularly in gathering high-quality data and designing
comprehensive model frameworks. To bridge the gap in pathology MLLMs, we
present PathAsst, a multimodal generative foundation AI assistant to
revolutionize diagnostic and predictive analytics in pathology. The development
of PathAsst involves three pivotal steps: data acquisition, CLIP model
adaptation, and the training of PathAsst's multimodal generative capabilities.
Firstly, we collect over 207K high-quality pathology image-text pairs from
authoritative sources. Leveraging the advanced power of ChatGPT, we generate
over 180K instruction-following samples. Furthermore, we devise additional
instruction-following data specifically tailored for invoking eight
pathology-specific sub-models we prepared, allowing the PathAsst to effectively
collaborate with these models, enhancing its diagnostic ability. Secondly, by
leveraging the collected data, we construct PathCLIP, a pathology-dedicated
CLIP, to enhance PathAsst's capabilities in interpreting pathology images.
Finally, we integrate PathCLIP with the Vicuna-13b and utilize
pathology-specific instruction-tuning data to enhance the multimodal generation
capacity of PathAsst and bolster its synergistic interactions with sub-models.
The experimental results of PathAsst show the potential of harnessing
AI-powered generative foundation model to improve pathology diagnosis and
treatment processes.
