  Recent works have demonstrated that using reinforcement learning (RL) with
multiple quality rewards can improve the quality of generated images in
text-to-image (T2I) generation. However, manually adjusting reward weights
poses challenges and may cause over-optimization in certain metrics. To solve
this, we propose Parrot, which addresses the issue through multi-objective
optimization and introduces an effective multi-reward optimization strategy to
approximate Pareto optimal. Utilizing batch-wise Pareto optimal selection,
Parrot automatically identifies the optimal trade-off among different rewards.
We use the novel multi-reward optimization algorithm to jointly optimize the
T2I model and a prompt expansion network, resulting in significant improvement
of image quality and also allow to control the trade-off of different rewards
using a reward related prompt during inference. Furthermore, we introduce
original prompt-centered guidance at inference time, ensuring fidelity to user
input after prompt expansion. Extensive experiments and a user study validate
the superiority of Parrot over several baselines across various quality
criteria, including aesthetics, human preference, text-image alignment, and
image sentiment.
