  This paper introduces a novel unified representation of diffusion models for
image generation and segmentation. Specifically, we use a colormap to represent
entity-level masks, addressing the challenge of varying entity numbers while
aligning the representation closely with the image RGB domain. Two novel
modules, including the location-aware color palette and progressive dichotomy
module, are proposed to support our mask representation. On the one hand, a
location-aware palette guarantees the colors' consistency to entities'
locations. On the other hand, the progressive dichotomy module can efficiently
decode the synthesized colormap to high-quality entity-level masks in a
depth-first binary search without knowing the cluster numbers. To tackle the
issue of lacking large-scale segmentation training data, we employ an
inpainting pipeline and then improve the flexibility of diffusion models across
various tasks, including inpainting, image synthesis, referring segmentation,
and entity segmentation. Comprehensive experiments validate the efficiency of
our approach, demonstrating comparable segmentation mask quality to
state-of-the-art and adaptability to multiple tasks. The code will be released
at \href{https://github.com/qqlu/Entity}{https://github.com/qqlu/Entity}.
