  Text-to-image generative models have recently attracted considerable
interest, enabling the synthesis of high-quality images from textual prompts.
However, these models often lack the capability to generate specific subjects
from given reference images or to synthesize novel renditions under varying
conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTI)
have made significant progress in this area. Yet, both approaches primarily
focus on enhancing similarity to reference images and require expensive setups,
often overlooking the need for efficient training and avoiding overfitting to
the reference images. In this work, we present the $\lambda$-Harmonic reward
function, which provides a reliable reward signal and enables early stopping
for faster training and effective regularization. By combining the
Bradley-Terry preference model, the $\lambda$-Harmonic reward function also
provides preference labels for subject-driven generation tasks. We propose
Reward Preference Optimization (RPO), which offers a simpler setup (requiring
only $3\%$ of the negative samples used by DreamBooth) and fewer gradient steps
for fine-tuning. Unlike most existing methods, our approach does not require
training a text encoder or optimizing text embeddings and achieves text-image
alignment by fine-tuning only the U-Net component. Empirically,
$\lambda$-Harmonic proves to be a reliable approach for model selection in
subject-driven generation tasks. Based on preference labels and early stopping
validation from the $\lambda$-Harmonic reward function, our algorithm achieves
a state-of-the-art CLIP-I score of 0.833 and a CLIP-T score of 0.314 on
DreamBench.
