  Conditional diffusion models have demonstrated impressive performance on
various tasks like text-guided semantic image editing. Prior work requires
image regions to be identified manually by human users or use an object
detector that only perform well for object-centric manipulations. For example,
if an input image contains multiple objects with the same semantic meaning
(such as a group of birds), object detectors may struggle to recognize and
localize the target object, let alone accurately manipulate it. To address
these challenges, we propose a two-stage method for achieving complex scene
image editing by Scene Graph Comprehension (SGC-Net). In the first stage, we
train a Region of Interest (RoI) prediction network that uses scene graphs and
predict the locations of the target objects. Unlike object detection methods
based solely on object category, our method can accurately recognize the target
object by comprehending the objects and their semantic relationships within a
complex scene. The second stage uses a conditional diffusion model to edit the
image based on our RoI predictions. We evaluate the effectiveness of our
approach on the CLEVR and Visual Genome datasets. We report an 8 point
improvement in SSIM on CLEVR and our edited images were preferred by human
users by 9-33% over prior work on Visual Genome, validating the effectiveness
of our proposed method. Code is available at
github.com/Zhongping-Zhang/SGC_Net.
