  Reinforcement learning from human feedback (RLHF) is a prevalent approach to
align AI systems with human values by learning rewards from human preference
data. Due to various reasons, however, such data typically takes the form of
rankings over pairs of trajectory segments, which fails to capture the varying
strengths of preferences across different pairs. In this paper, we propose a
novel adaptive preference loss, underpinned by distributionally robust
optimization (DRO), designed to address this uncertainty in preference
strength. By incorporating an adaptive scaling parameter into the loss for each
pair, our method increases the flexibility of the reward function.
Specifically, it assigns small scaling parameters to pairs with ambiguous
preferences, leading to more comparable rewards, and large scaling parameters
to those with clear preferences for more distinct rewards. Computationally, our
proposed loss function is strictly convex and univariate with respect to each
scaling parameter, enabling its efficient optimization through a simple
second-order algorithm. Our method is versatile and can be readily adapted to
various preference optimization frameworks, including direct preference
optimization (DPO). Our experiments with robotic control and natural language
generation with large language models (LLMs) show that our method not only
improves policy performance but also aligns reward function selection more
closely with policy optimization, simplifying the hyperparameter tuning
process.
