  Language model alignment methods, such as reinforcement learning from human
feedback (RLHF), have led to impressive advances in language model
capabilities, but existing techniques are limited by a widely observed
phenomenon known as overoptimization, where the quality of the language model
plateaus or degrades over the course of the alignment process. Overoptimization
is often attributed to overfitting to an inaccurate reward model, and while it
can be mitigated through online data collection, this is infeasible in many
settings. This raises a fundamental question: Do existing offline alignment
algorithms make the most of the data they have, or can their sample-efficiency
be improved further?
  We address this question with a new algorithm for offline alignment,
$\chi^2$-Preference Optimization ($\chi$PO). $\chi$PO is a one-line change to
Direct Preference Optimization (DPO; Rafailov et al., 2023), which only
involves modifying the logarithmic link function in the DPO objective. Despite
this minimal change, $\chi$PO implicitly implements the principle of pessimism
in the face of uncertainty via regularization with the $\chi^2$-divergence --
which quantifies uncertainty more effectively than KL-regularization -- and
provably alleviates overoptimization, achieving sample-complexity guarantees
based on single-policy concentrability -- the gold standard in offline
reinforcement learning. $\chi$PO's simplicity and strong guarantees make it the
first practical and general-purpose offline alignment algorithm that is
provably robust to overoptimization.
