  The alignment of language models with human preferences is vital for their
application in real-world tasks. The problem is formulated as optimizing the
model's policy to maximize the expected reward that reflects human preferences
with minimal deviation from the initial policy. While considered as a
straightforward solution, reinforcement learning (RL) suffers from high
variance in policy updates, which impedes efficient policy improvement.
Recently, direct preference optimization (DPO) was proposed to directly
optimize the policy from preference data. However, we show that DPO derived
based on the optimal solution of the problem leads to a compromised
mean-seeking approximation of the optimal solution in practice. In this paper,
we propose efficient exact optimization (EXO) of the alignment objective. EXO
is guaranteed to optimize in the same direction as RL algorithms asymptotically
for arbitrary policy parametrization. This leads to the same mode-seeking
solution, while enables efficient optimization by circumventing the
complexities of RL. We also compare our method to DPO with both theoretical and
empirical analyses, and further demonstrate the advantages of our method over
existing approaches on realistic human preference data. Code is available at
https://github.com/haozheji/exact-optimization.
