  Large language models (LLMs) have revolutionized the role of AI, yet also
pose potential risks of propagating unethical content. Alignment technologies
have been introduced to steer LLMs towards human preference, gaining increasing
attention. Despite notable breakthroughs in this direction, existing methods
heavily rely on high-quality positive-negative training pairs, suffering from
noisy labels and the marginal distinction between preferred and dispreferred
response data. Given recent LLMs' proficiency in generating helpful responses,
this work pivots towards a new research focus: achieving alignment using solely
human-annotated negative samples, preserving helpfulness while reducing
harmfulness. For this purpose, we propose Distributional Dispreference
Optimization (D$^2$O), which maximizes the discrepancy between the generated
responses and the dispreferred ones to effectively eschew harmful information.
We theoretically demonstrate that D$^2$O is equivalent to learning a
distributional instead of instance-level preference model reflecting human
dispreference against the distribution of negative responses. Besides, D$^2$O
integrates an implicit Jeffrey Divergence regularization to balance the
exploitation and exploration of reference policies and converges to a
non-negative one during training. Extensive experiments demonstrate that our
method achieves comparable generation quality and surpasses the latest
baselines in producing less harmful and more informative responses with better
training stability and faster convergence.
