  Learning from human feedback has been shown to improve text-to-image models.
These techniques first learn a reward function that captures what humans care
about in the task and then improve the models based on the learned reward
function. Even though relatively simple approaches (e.g., rejection sampling
based on reward scores) have been investigated, fine-tuning text-to-image
models with the reward function remains challenging. In this work, we propose
using online reinforcement learning (RL) to fine-tune text-to-image models. We
focus on diffusion models, defining the fine-tuning task as an RL problem, and
updating the pre-trained text-to-image diffusion models using policy gradient
to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
policy optimization with KL regularization. We conduct an analysis of KL
regularization for both RL fine-tuning and supervised fine-tuning. In our
experiments, we show that DPOK is generally superior to supervised fine-tuning
with respect to both image-text alignment and image quality. Our code is
available at
https://github.com/google-research/google-research/tree/master/dpok.
