  Diffusion models have shown impressive performance in many domains, including
image generation, time series prediction, and reinforcement learning. The
algorithm demonstrates superior performance over the traditional GAN and
transformer-based methods. However, the model's capability to follow natural
language instructions (e.g., spatial relationships between objects, generating
complex scenes) is still unsatisfactory. It has been an important research area
to enhance such capability. Prior works have shown that using Reinforcement
Learning can effectively train diffusion models to enhance fidelity on specific
objectives. However, existing RL methods require collecting a large amount of
data to train an effective reward model. They also don't receive feedback when
the generated image is incorrect. In this work, we propose Iterative Prompt
Relabeling (IPR), a novel algorithm that aligns images to text through
iterative image sampling and prompt relabeling. IPR first samples a batch of
images conditioned on the text then relabels the text prompts of unmatched
text-image pairs with classifier feedback. We conduct thorough experiments on
SDv2 and SDXL, testing their capability to follow instructions on spatial
relations. With IPR, we improved up to 15.22% (absolute improvement) on the
challenging spatial relation VISOR benchmark, demonstrating superior
performance compared to previous RL methods.
