  This study addresses the challenge of noise in training datasets for Direct
Preference Optimization (DPO), a method for aligning Large Language Models
(LLMs) with human preferences. We categorize noise into pointwise noise, which
includes low-quality data points, and pairwise noise, which encompasses
erroneous data pair associations that affect preference rankings. Utilizing
Distributionally Robust Optimization (DRO), we enhance DPO's resilience to
these types of noise. Our theoretical insights reveal that DPO inherently
embeds DRO principles, conferring robustness to pointwise noise, with the
regularization coefficient $\beta$ playing a critical role in its noise
resistance. Extending this framework, we introduce Distributionally
Robustifying DPO (Dr. DPO), which integrates pairwise robustness by optimizing
against worst-case pairwise scenarios. The novel hyperparameter $\beta'$ in Dr.
DPO allows for fine-tuned control over data pair reliability, providing a
strategic balance between exploration and exploitation in noisy training
environments. Empirical evaluations demonstrate that Dr. DPO substantially
improves the quality of generated text and response accuracy in preference
datasets, showcasing enhanced performance in both noisy and noise-free
settings. The code is available at https://github.com/junkangwu/Dr_DPO.
