  Training Large Language Models (LLMs) with Reinforcement Learning from AI
Feedback (RLAIF) aligns model outputs more closely with human preferences. This
involves an evaluator model ranking multiple candidate responses to user
prompts. However, the rankings from popular evaluator models such as GPT-4 can
be inconsistent. We propose the Repeat Ranking method - where we evaluate the
same responses multiple times and train only on those responses which are
consistently ranked. Using 2,714 prompts in 62 languages, we generated
responses from 7 top multilingual LLMs and had GPT-4 rank them five times each.
Evaluating on MT-Bench chat benchmarks in six languages, our method
outperformed the standard practice of training on all available prompts. Our
work highlights the quality versus quantity trade-off in RLAIF dataset
generation and offers a stackable strategy for enhancing dataset and thus model
quality.
