  Text-conditioned image generation has made significant progress in recent
years with generative adversarial networks and more recently, diffusion models.
While diffusion models conditioned on text prompts have produced impressive and
high-quality images, accurately representing complex text prompts such as the
number of instances of a specific object remains challenging.
  To address this limitation, we propose a novel guidance approach for the
sampling process in the diffusion model that leverages bounding box and
segmentation map information at inference time without additional training
data. Through a novel loss in the sampling process, our approach guides the
model with semantic features from CLIP embeddings and enforces geometric
constraints, leading to high-resolution images that accurately represent the
scene. To obtain bounding box and segmentation map information, we structure
the text prompt as a scene graph and enrich the nodes with CLIP embeddings. Our
proposed model achieves state-of-the-art performance on two public benchmarks
for image generation from scene graphs, surpassing both scene graph to image
and text-based diffusion models in various metrics. Our results demonstrate the
effectiveness of incorporating bounding box and segmentation map guidance in
the diffusion model sampling process for more accurate text-to-image
generation.
