  This work studies the challenge of aligning large language models (LLMs) with
offline preference data. We focus on alignment by Reinforcement Learning from
Human Feedback (RLHF) in particular. While popular preference optimization
methods exhibit good empirical performance in practice, they are not
theoretically guaranteed to converge to the optimal policy and can provably
fail when the data coverage is sparse by classical offline reinforcement
learning (RL) results. On the other hand, a recent line of work has focused on
theoretically motivated preference optimization methods with provable
guarantees, but these are not computationally efficient for large-scale
applications like LLM alignment. To bridge this gap, we propose SPAC, a new
offline preference optimization method with self-play, inspired by the
on-average pessimism technique from the offline RL literature, to be the first
provable and scalable approach to LLM alignment. We both provide theoretical
analysis for its convergence under single-policy concentrability for the
general function approximation setting and demonstrate its competitive
empirical performance for LLM alignment on a 7B Mistral model with Open LLM
Leaderboard evaluations.
