  Learning from preference feedback has emerged as an essential step for
improving the generation quality and performance of modern language models
(LMs). Despite its widespread use, the way preference-based learning is applied
varies wildly, with differing data, learning algorithms, and evaluations used,
making disentangling the impact of each aspect difficult. In this work, we
identify four core aspects of preference-based learning: preference data,
learning algorithm, reward model, and policy training prompts, systematically
investigate the impact of these components on downstream model performance, and
suggest a recipe for strong learning for preference feedback. Our findings
indicate that all aspects are important for performance, with better preference
data leading to the largest improvements, followed by the choice of learning
algorithm, the use of improved reward models, and finally the use of additional
unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to
2.5% in math and 1.2% in general domains. High-quality preference data leads to
improvements of up to 8% in instruction following and truthfulness. Despite
significant gains of up to 5% in mathematical evaluation when scaling up reward
models, we surprisingly observe marginal improvements in other categories.
  We publicly release the code used for training
(https://github.com/hamishivi/EasyLM) and evaluating
(https://github.com/allenai/open-instruct) our models, along with the models
and datasets themselves
(https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).
