  Fine-grained control over large language models (LLMs) remains a significant
challenge, hindering their adaptability to diverse user needs. While
Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning
LLMs, its reliance on scalar rewards often limits its ability to capture
diverse user preferences in real-world applications. To address this
limitation, we introduce the Directional Preference Alignment (DPA) framework.
Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling
to represent diverse preference profiles. Additionally, DPA models user
preferences as directions (i.e., unit vectors) in the reward space to achieve
user-dependent preference control. Our method involves training a
multi-objective reward model and then fine-tuning the LLM with a
preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF
method adopted by Llama 2. This method enjoys a better performance trade-off
across various reward objectives. In comparison with the scalar-reward RLHF,
DPA offers users intuitive control over LLM generation: they can arithmetically
specify their desired trade-offs (e.g., more helpfulness with less verbosity).
We also validate the effectiveness of DPA with real-world alignment experiments
on Mistral-7B. Our method provides straightforward arithmetic control over the
trade-off between helpfulness and verbosity while maintaining competitive
performance with strong baselines such as Direct Preference Optimization (DPO).
