  Reinforcement learning from human feedback (RLHF) is a promising solution to
align large language models (LLMs) more closely with human values. Off-policy
preference optimization, where the preference data is obtained from other
models, is widely adopted due to its cost efficiency and scalability. However,
off-policy preference optimization often suffers from a distributional gap
between the policy used for data collection and the target policy, leading to
suboptimal optimization. In this paper, we propose a novel strategy to mitigate
this problem by simulating on-policy learning with off-policy preference data.
Our Weighted Preference Optimization (WPO) method adapts off-policy data to
resemble on-policy data more closely by reweighting preference pairs according
to their probability under the current policy. This method not only addresses
the distributional gap problem but also enhances the optimization process
without incurring additional costs. We validate our method on instruction
following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
but also establishes a remarkable length-controlled winning rate against
GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
model on the leaderboard. We will release the code and models at
https://github.com/wzhouad/WPO.
