  Diffusion models have demonstrated impressive performance in text-guided
image generation. Current methods that leverage the knowledge of these models
for image editing either fine-tune them using the input image (e.g., Imagic) or
incorporate structure information as additional constraints (e.g., ControlNet).
However, fine-tuning large-scale diffusion models on a single image can lead to
severe overfitting issues and lengthy inference time. Information leakage from
pretrained models also make it challenging to preserve image content not
related to the text input. Additionally, methods that incorporate structural
guidance (e.g., edge maps, semantic maps, keypoints) find retaining attributes
like colors and textures difficult. Using the input image as a control could
mitigate these issues, but since these models are trained via reconstruction, a
model can simply hide information about the original image when encoding it to
perfectly reconstruct the image without learning the editing task. To address
these challenges, we propose a text-to-image editing model with an Image
Information Removal module (IIR) that selectively erases color-related and
texture-related information from the original image, allowing us to better
preserve the text-irrelevant content and avoid issues arising from information
hiding. Our experiments on CUB, Outdoor Scenes, and COCO reports our approach
achieves the best editability-fidelity trade-off results. In addition, a user
study on COCO shows that our edited images are preferred 35% more often than
prior work.
