  Direct Preference Optimization (DPO) has recently emerged as a popular
approach to improve reinforcement learning with human feedback (RLHF), leading
to better techniques to fine-tune large language models (LLM). A weakness of
DPO, however, lies in its lack of capability to characterize the diversity of
human preferences. Inspired by Mallows' theory of preference ranking, we
develop in this paper a new approach, the Mallows-DPO. A distinct feature of
this approach is a dispersion index, which reflects the dispersion of human
preference to prompts. We show that existing DPO models can be reduced to
special cases of this dispersion index, thus unified with Mallows-DPO. More
importantly, we demonstrate (empirically) how to use this dispersion index to
enhance the performance of DPO in a broad array of benchmark tasks, from
synthetic bandit selection to controllable generations and dialogues, while
maintaining great generalization capabilities.
