  Preference datasets are essential for incorporating human preferences into
pre-trained language models, playing a key role in the success of Reinforcement
Learning from Human Feedback. However, these datasets often demonstrate
conflicting alignment objectives, leading to increased vulnerability to
jailbreak attacks and challenges in adapting downstream tasks to prioritize
specific alignment objectives without negatively impacting others. In this
work, we introduce a novel statistical metric, Alignment Dimension Conflict, to
quantify the degree of conflict within preference datasets. We then present
\texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative
pairwise preference datasets with reduced-conflict alignment objectives.
\texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback
from GPT-4, marking as the first preference dataset aimed at reducing the
competition between alignment objectives. Furthermore, we develop reward
models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to
balance diverse alignment objectives effectively. This sampling method
positions HummerRM as an ideal model for domain-specific further fine-tuning
and reducing vulnerabilities to attacks.
