  AI in dermatology is evolving at a rapid pace but the major limitation to
training trustworthy classifiers is the scarcity of data with ground-truth
concept level labels, which are meta-labels semantically meaningful to humans.
Foundation models like CLIP providing zero-shot capabilities can help alleviate
this challenge by leveraging vast amounts of image-caption pairs available on
the internet. CLIP can be fine-tuned using domain specific image-caption pairs
to improve classification performance. However, CLIP's pre-training data is not
well-aligned with the medical jargon that clinicians use to perform diagnoses.
The development of large language models (LLMs) in recent years has led to the
possibility of leveraging the expressive nature of these models to generate
rich text. Our goal is to use these models to generate caption text that aligns
well with both the clinical lexicon and with the natural human language used in
CLIP's pre-training data. Starting with captions used for images in PubMed
articles, we extend them by passing the raw captions through an LLM fine-tuned
on the field's several textbooks. We find that using captions generated by an
expressive fine-tuned LLM like GPT-3.5 improves downstream zero-shot concept
classification performance.
