  Preference learning algorithms (e.g., RLHF and DPO) are frequently used to
steer LLMs to produce generations that are more preferred by humans, but our
understanding of their inner workings is still limited. In this work, we study
the conventional wisdom that preference learning trains models to assign higher
likelihoods to more preferred outputs than less preferred outputs, measured via
$\textit{ranking accuracy}$. Surprisingly, we find that most state-of-the-art
preference-tuned models achieve a ranking accuracy of less than 60% on common
preference datasets. We furthermore derive the $\textit{idealized ranking
accuracy}$ that a preference-tuned LLM would achieve if it optimized the DPO or
RLHF objective perfectly. We demonstrate that existing models exhibit a
significant $\textit{alignment gap}$ -- $\textit{i.e.}$, a gap between the
observed and idealized ranking accuracies. We attribute this discrepancy to the
DPO objective, which is empirically and theoretically ill-suited to fix even
mild ranking errors in the reference model, and derive a simple and efficient
formula for quantifying the difficulty of learning a given preference
datapoint. Finally, we demonstrate that ranking accuracy strongly correlates
with the empirically popular win rate metric when the model is close to the
reference model used in the objective, shedding further light on the
differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO)
preference learning algorithms.
