  Reinforcement Learning from Human Feedback (RLHF) is commonly used to
fine-tune large language models to better align with human preferences.
However, the underlying premise of algorithms developed under this framework
can be problematic when user preferences encoded in human feedback are diverse.
In this work, we aim to address this problem by developing methods for building
personalized language models. We first formally introduce the task of learning
from personalized human feedback and explain why vanilla RLHF can be
ineffective in this context. We then propose a general Personalized-RLHF
(P-RLHF) framework, including a user model that maps user information to user
representations and can flexibly encode our assumptions on user preferences. We
develop new learning objectives to perform personalized Direct Preference
Optimization that jointly learns a user model and a personalized language
model. We demonstrate the efficacy of our proposed method through (1) a
synthetic task where we fine-tune a GPT-J 6B model to align with users with
conflicting preferences on generation length; and (2) an instruction following
task where we fine-tune a Tulu-7B model to generate responses for users with
diverse preferences on the style of responses. In both cases, our learned
models can generate personalized responses that are better aligned with the
preferences of individual users.
