  Aside from offering state-of-the-art performance in medical image generation,
denoising diffusion probabilistic models (DPM) can also serve as a
representation learner to capture semantic information and potentially be used
as an image representation for downstream tasks, e.g., segmentation. However,
these latent semantic representations rely heavily on labor-intensive
pixel-level annotations as supervision, limiting the usability of DPM in
medical image segmentation. To address this limitation, we propose an enhanced
diffusion segmentation model, called TextDiff, that improves semantic
representation through inexpensive medical text annotations, thereby explicitly
establishing semantic representation and language correspondence for diffusion
models. Concretely, TextDiff extracts intermediate activations of the Markov
step of the reverse diffusion process in a pretrained diffusion model on
large-scale natural images and learns additional expert knowledge by combining
them with complementary and readily available diagnostic text information.
TextDiff freezes the dual-branch multi-modal structure and mines the latent
alignment of semantic features in diffusion models with diagnostic descriptions
by only training the cross-attention mechanism and pixel classifier, making it
possible to enhance semantic representation with inexpensive text. Extensive
experiments on public QaTa-COVID19 and MoNuSeg datasets show that our TextDiff
is significantly superior to the state-of-the-art multi-modal segmentation
methods with only a few training samples.
