  Multi-modal foundation models are typically trained on millions of pairs of
natural images and text captions, frequently obtained through web-crawling
approaches. Although such models depict excellent generative capabilities, they
do not typically generalize well to specific domains such as medical images
that have fundamentally shifted distributions compared to natural images.
Building generative models for medical images that faithfully depict clinical
context may help alleviate the paucity of healthcare datasets. Thus, in this
study, we seek to research and expand the representational capabilities of
large pretrained foundation models to medical concepts, specifically for
leveraging the Stable Diffusion model to generate domain specific images found
in medical imaging. We explore the sub-components of the Stable Diffusion
pipeline (the variational autoencoder, the U-Net and the text-encoder) to
fine-tune the model to generate medical images. We benchmark the efficacy of
these efforts using quantitative image quality metrics and qualitative
radiologist-driven evaluations that accurately represent the clinical content
of conditional text prompts. Our best-performing model improves upon the stable
diffusion baseline and can be conditioned to insert a realistic-looking
abnormality on a synthetic radiology image, while maintaining a 95% accuracy on
a classifier trained to detect the abnormality.
