  This paper studies the alignment process of generative models with
Reinforcement Learning from Human Feedback (RLHF). We first identify the
primary challenges of existing popular methods like offline PPO and offline DPO
as lacking in strategical exploration of the environment. Then, to understand
the mathematical principle of RLHF, we consider a standard mathematical
formulation, the reverse-KL regularized contextual bandit for RLHF. Despite its
widespread practical application, a rigorous theoretical analysis of this
formulation remains open. We investigate its behavior in three distinct
settings -- offline, online, and hybrid -- and propose efficient algorithms
with finite-sample theoretical guarantees.
  Moving towards practical applications, our framework, with a robust
approximation of the information-theoretical policy improvement oracle,
naturally gives rise to several novel RLHF algorithms. This includes an
iterative version of the Direct Preference Optimization (DPO) algorithm for
online settings, and a multi-step rejection sampling strategy for offline
scenarios. Our empirical evaluations on real-world alignment experiment of
large language model demonstrate that these proposed methods significantly
surpass existing strong baselines, such as DPO and Rejection Sampling
Optimization (RSO), showcasing the connections between solid theoretical
foundations and their potent practical implementations.
