  Reinforcement learning from human feedback (RLHF) has proven effective in
aligning large language models (LLMs) with human preferences. However,
gathering high-quality human preference labels can be a time-consuming and
expensive endeavor. RL from AI Feedback (RLAIF), introduced by Bai et al.,
offers a promising alternative that leverages a powerful off-the-shelf LLM to
generate preferences in lieu of human annotators. Across the tasks of
summarization, helpful dialogue generation, and harmless dialogue generation,
RLAIF achieves comparable or superior performance to RLHF, as rated by human
evaluators. Furthermore, RLAIF demonstrates the ability to outperform a
supervised fine-tuned baseline even when the LLM preference labeler is the same
size as the policy. In another experiment, directly prompting the LLM for
reward scores achieves superior performance to the canonical RLAIF setup, where
LLM preference labels are first distilled into a reward model. Finally, we
conduct extensive studies on techniques for generating aligned AI preferences.
Our results suggest that RLAIF can achieve human-level performance, offering a
potential solution to the scalability limitations of RLHF.
