  Reinforcement learning from human feedback (RLHF) has demonstrated great
promise in aligning large language models (LLMs) with human preference.
Depending on the availability of preference data, both online and offline RLHF
are active areas of investigation. A key bottleneck is understanding how to
incorporate uncertainty estimation in the reward function learned from the
preference data for RLHF, regardless of how the preference data is collected.
While the principles of optimism or pessimism under uncertainty are
well-established in standard reinforcement learning (RL), a
practically-implementable and theoretically-grounded form amenable to large
language models is not yet available, as standard techniques for constructing
confidence intervals become intractable under arbitrary policy
parameterizations.
  In this paper, we introduce a unified approach to online and offline RLHF --
value-incentivized preference optimization (VPO) -- which regularizes the
maximum-likelihood estimate of the reward function with the corresponding value
function, modulated by a $\textit{sign}$ to indicate whether the optimism or
pessimism is chosen. VPO also directly optimizes the policy with implicit
reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
preference optimization. Theoretical guarantees of VPO are provided for both
online and offline settings, matching the rates of their standard RL
counterparts. Moreover, experiments on text summarization and dialog verify the
practicality and effectiveness of VPO.
