  Learning from human preference data has emerged as the dominant paradigm for
fine-tuning large language models (LLMs). The two most common families of
techniques -- online reinforcement learning (RL) such as Proximal Policy
Optimization (PPO) and offline contrastive methods such as Direct Preference
Optimization (DPO) -- were positioned as equivalent in prior work due to the
fact that both have to start from the same offline preference dataset. To
further expand our theoretical understanding of the similarities and
differences between online and offline techniques for preference fine-tuning,
we conduct a rigorous analysis through the lens of dataset coverage, a concept
that captures how the training data covers the test distribution and is widely
used in RL. We prove that a global coverage condition is both necessary and
sufficient for offline contrastive methods to converge to the optimal policy,
but a weaker partial coverage condition suffices for online RL methods. This
separation provides one explanation of why online RL methods can perform better
than offline methods, especially when the offline preference data is not
diverse enough. Finally, motivated by our preceding theoretical observations,
we derive a hybrid preference optimization (HyPO) algorithm that uses offline
data for contrastive-based preference optimization and online data for KL
regularization. Theoretically and empirically, we demonstrate that HyPO is more
performant than its pure offline counterpart DPO, while still preserving its
computation and memory efficiency.
