  Reinforcement learning from human feedback (RLHF) has emerged as a powerful
technique to make large language models (LLMs) more capable in complex
settings. RLHF proceeds as collecting human preference data, training a reward
model on said data, and optimizing a base ML model with respect to said reward
for extrinsic evaluation metrics (e.g. MMLU, GSM8k). RLHF relies on many
assumptions about how the various pieces fit together, such as a reward model
capturing human preferences and an RL optimizer extracting the right signal
from a reward model. As the RLHF process involves many distinct design
decisions, it is easy to assume that multiple processes are correlated and
therefore numerically linked. This apparent correlation is often not true,
where reward models are easily overoptimized or RL optimizers can reduce
performance on tasks not modeled in the data. Notable manifestations of models
trained with imperfect RLHF systems are those that are prone to refusing basic
requests for safety reasons or appearing lazy in generations. As chat model
evaluation becomes increasingly nuanced, the reliance on a perceived link
between reward model training, RL scores, and downstream performance drives
these issues, which we describe as an objective mismatch. In this paper, we
illustrate the causes of this issue, reviewing relevant literature from
model-based reinforcement learning, and argue for solutions. By solving
objective mismatch in RLHF, the ML models of the future will be more precisely
aligned to user instructions for both safety and helpfulness.
