  Aligning language models with preferences can be posed as approximating a
target distribution representing some desired behavior. Existing approaches
differ both in the functional form of the target distribution and the algorithm
used to approximate it. For instance, Reinforcement Learning from Human
Feedback (RLHF) corresponds to minimizing a reverse KL from an implicit target
distribution arising from a KL penalty in the objective. On the other hand,
Generative Distributional Control (GDC) has an explicit target distribution and
minimizes a forward KL from it using the Distributional Policy Gradient (DPG)
algorithm. In this paper, we propose a new approach, f-DPG, which allows the
use of any f-divergence to approximate any target distribution that can be
evaluated. f-DPG unifies both frameworks (RLHF, GDC) and the approximation
methods (DPG, RL with KL penalties). We show the practical benefits of various
choices of divergence objectives and demonstrate that there is no universally
optimal objective but that different divergences present different alignment
and diversity trade-offs. We show that Jensen-Shannon divergence strikes a good
balance between these objectives, and frequently outperforms forward KL
divergence by a wide margin, leading to significant improvements over prior
work. These distinguishing characteristics between divergences persist as the
model size increases, highlighting the importance of selecting appropriate
divergence objectives.
