  Large foundation models pretrained on raw web-scale data are not readily
deployable without additional step of extensive alignment to human preferences.
Such alignment is typically done by collecting large amounts of pairwise
comparisons from humans ("Do you prefer output A or B?") and learning a reward
model or a policy with the Bradley-Terry-Luce (BTL) model as a proxy for a
human's underlying implicit preferences. These methods generally suffer from
assuming a universal preference shared by all humans, which lacks the
flexibility of adapting to plurality of opinions and preferences. In this work,
we propose PAL, a framework to model human preference complementary to existing
pretraining strategies, which incorporates plurality from the ground up. We
propose using the ideal point model as a lens to view alignment using
preference comparisons. Together with our novel reformulation and using mixture
modeling, our framework captures the plurality of population preferences while
simultaneously learning a common preference latent space across different
preferences, which can few-shot generalize to new, unseen users. Our approach
enables us to use the penultimate-layer representation of large foundation
models and simple MLP layers to learn reward functions that are on-par with the
existing large state-of-the-art reward models, thereby enhancing efficiency of
reward modeling significantly. We show that PAL achieves competitive reward
model accuracy compared to strong baselines on 1) Language models with Summary
dataset ; 2) Image Generative models with Pick-a-Pic dataset ; 3) A new
semisynthetic heterogeneous dataset generated using Anthropic Personas.
Finally, our experiments also highlight the shortcoming of current preference
datasets that are created using rigid rubrics which wash away heterogeneity,
and call for more nuanced data collection approaches.
