  The pre-trained text-image discriminative models, such as CLIP, has been
explored for open-vocabulary semantic segmentation with unsatisfactory results
due to the loss of crucial localization information and awareness of object
shapes. Recently, there has been a growing interest in expanding the
application of generative models from generation tasks to semantic
segmentation. These approaches utilize generative models either for generating
annotated data or extracting features to facilitate semantic segmentation. This
typically involves generating a considerable amount of synthetic data or
requiring additional mask annotations. To this end, we uncover the potential of
generative text-to-image diffusion models (e.g., Stable Diffusion) as highly
efficient open-vocabulary semantic segmenters, and introduce a novel
training-free approach named DiffSegmenter. The insight is that to generate
realistic objects that are semantically faithful to the input text, both the
complete object shapes and the corresponding semantics are implicitly learned
by diffusion models. We discover that the object shapes are characterized by
the self-attention maps while the semantics are indicated through the
cross-attention maps produced by the denoising U-Net, forming the basis of our
segmentation results.Additionally, we carefully design effective textual
prompts and a category filtering mechanism to further enhance the segmentation
results. Extensive experiments on three benchmark datasets show that the
proposed DiffSegmenter achieves impressive results for open-vocabulary semantic
segmentation.
