  Existing text-guided image manipulation methods aim to modify the appearance
of the image or to edit a few objects in a virtual or simple scenario, which is
far from practical applications. In this work, we study a novel task on
text-guided image manipulation on the entity level in the real world (eL-TGIM).
The task imposes three basic requirements, (1) to edit the entity consistent
with the text descriptions, (2) to preserve the entity-irrelevant regions, and
(3) to merge the manipulated entity into the image naturally. To this end, we
propose an elegant framework, dubbed as SeMani, forming the Semantic
Manipulation of real-world images that can not only edit the appearance of
entities but also generate new entities corresponding to the text guidance. To
solve eL-TGIM, SeMani decomposes the task into two phases: the semantic
alignment phase and the image manipulation phase. In the semantic alignment
phase, SeMani incorporates a semantic alignment module to locate the
entity-relevant region to be manipulated. In the image manipulation phase,
SeMani adopts a generative model to synthesize new images conditioned on the
entity-irrelevant regions and target text descriptions. We discuss and propose
two popular generation processes that can be utilized in SeMani, the discrete
auto-regressive generation with transformers and the continuous denoising
generation with diffusion models, yielding SeMani-Trans and SeMani-Diff,
respectively. We conduct extensive experiments on the real datasets CUB,
Oxford, and COCO datasets to verify that SeMani can distinguish the
entity-relevant and -irrelevant regions and achieve more precise and flexible
manipulation in a zero-shot manner compared with baseline methods. Our codes
and models will be released at https://github.com/Yikai-Wang/SeMani.
