  With the growing utilization of large language models (LLMs) across domains,
alignment towards human preferences has become one of the most critical aspects
of training models. At the forefront of state-of-the-art human alignment
methods are preference optimization methods (*PO). However, prior research has
often concentrated on identifying the best-performing method, typically
involving a grid search over hyperparameters, which can be impractical for
general practitioners. In this paper, we aim to identify the algorithm that,
while being performant, is simultaneously more robust to varying
hyperparameters, thereby increasing the likelihood of achieving better results.
We focus on a realistic out-of-distribution (OOD) scenario that mirrors
real-world applications of human alignment, offering practical insights into
the strengths and weaknesses of these methods. Furthermore, to better
understand the shortcomings of generations from the different methods, we
analyze the model generations through the lens of KL divergence of the SFT
model and the response length statistics. Our analysis reveals that the widely
adopted DPO method consistently produces lengthy responses of inferior quality
that are very close to the SFT responses. Motivated by these findings, we
propose an embarrassingly simple extension to the DPO algorithm, LN-DPO,
resulting in more concise responses without sacrificing quality compared to the
policy obtained by vanilla DPO.
