  Aligning large language models with human preferences has emerged as a
critical focus in language modeling research. Yet, integrating preference
learning into Text-to-Image (T2I) generative models is still relatively
uncharted territory. The Diffusion-DPO technique made initial strides by
employing pairwise preference learning in diffusion models tailored for
specific text prompts. We introduce Diffusion-RPO, a new method designed to
align diffusion-based T2I models with human preferences more effectively. This
approach leverages both prompt-image pairs with identical prompts and those
with semantically related content across various modalities. Furthermore, we
have developed a new evaluation metric, style alignment, aimed at overcoming
the challenges of high costs, low reproducibility, and limited interpretability
prevalent in current evaluations of human preference alignment. Our findings
demonstrate that Diffusion-RPO outperforms established methods such as
Supervised Fine-Tuning and Diffusion-DPO in tuning Stable Diffusion versions
1.5 and XL-1.0, achieving superior results in both automated evaluations of
human preferences and style alignment. Our code is available at
https://github.com/yigu1008/Diffusion-RPO
