  This is the technique report for the winning solution of the CVPR2024 GenAI
Media Generation Challenge Workshop's Instruction-guided Image Editing track.
Instruction-guided image editing has been largely studied in recent years. The
most advanced methods, such as SmartEdit and MGIE, usually combine large
language models with diffusion models through joint training, where the former
provides text understanding ability, and the latter provides image generation
ability. However, in our experiments, we find that simply connecting large
language models and image generation models through intermediary guidance such
as masks instead of joint fine-tuning leads to a better editing performance and
success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided
Image Editing): editing category classification, main editing object
identification, editing mask acquisition, and image inpainting. Results show
that through proper combinations of language models and image inpainting
models, our pipeline can reach a high success rate with satisfying visual
quality.
