  Deep Reinforcement Learning is widely used for aligning Large Language Models
(LLM) with human preference. However, the conventional reward modelling is
predominantly dependent on human annotations provided by a select cohort of
individuals. Such dependence may unintentionally result in skewed models that
reflect the inclinations of these annotators, thereby failing to adequately
represent the wider population's expectations. We propose the Distributional
Preference Reward Model (DPRM), a simple yet effective framework to align large
language models with diverse human preferences. To this end, we characterize
multiple preferences by a categorical distribution and introduce a Bayesian
updater to accommodate shifted or new preferences. On top of that, we design an
optimal-transportation-based loss to calibrate DPRM to align with the
preference distribution. Finally, the expected reward is utilized to fine-tune
an LLM policy to generate responses favoured by the population. Our experiments
show that DPRM significantly enhances the alignment of LLMs with population
preference, yielding more accurate, unbiased, and contextually appropriate
responses.
