  The complexity of the alignment problem stems from the fact that existing
methods are considered unstable. Reinforcement Learning from Human Feedback
(RLHF) addresses this issue by minimizing the KL divergence between the trained
policy and the initial supervised fine-tuned policy (SFT) to avoid generating
out-of-domain samples for the reward model (RM). Recently, many methods have
emerged that shift from online to offline optimization, reformulating the RLHF
objective and removing the reward model (DPO, IPO, KTO). Despite eliminating
the reward model and the challenges it posed, these algorithms are still
constrained in terms of closeness of the trained policy to the SFT one. In our
paper, we argue that this implicit limitation in the offline optimization
methods leads to suboptimal results. To address this issue, we propose a class
of new methods called Trust Region (TR-DPO, TR-IPO, TR-KTO), which update the
reference policy during training. With this straightforward update approach, we
demonstrate the effectiveness of the new paradigm of language model alignment
against the classical one on the Anthropic-HH and Reddit TL;DR datasets. Most
notably, when automatically comparing TR methods and baselines side by side
using pretrained Pythia 6.9B models on the Reddit TL;DR task, the difference in
win rates reaches 8.4% for DPO, 14.3% for IPO, and 15% for KTO. Finally, by
assessing model response ratings grounded on criteria such as coherence,
correctness, helpfulness, and harmlessness, we demonstrate that our proposed
methods significantly outperform existing techniques.
