  A single language model (LM), despite aligning well with an average labeler
through reinforcement learning from human feedback (RLHF), may not universally
suit diverse human preferences. Recent approaches therefore opt for
customization by collecting multi-dimensional feedback and creating distinct
reward models (RMs) for each dimension (e.g., helpfulness, harmlessness, or
honesty). Different LMs can then be optimized for different preferences using
multi-objective RLHF (MORLHF) with different reward weightings. Yet, RL
fine-tuning is unstable and resource-heavy, especially for MORLHF with diverse
and usually conflicting objectives. In this paper, we present Multi-Objective
Direct Preference Optimization (MODPO), an RL-free algorithm that extends
Direct Preference Optimization (DPO) for multiple alignment objectives with
minimal overheads. Essentially, MODPO folds language modeling directly into
reward modeling, training LMs as implicit collective reward models (cRMs) that
combine all objectives with specific weightings. While theoretically guaranteed
to produce the same optimal solutions as MORLHF, MODPO is practically more
stable and computationally efficient. Empirical results from safety alignment
and long-form question answering confirm that MODPO matches or outperforms
existing methods, consistently producing a Pareto front of LMs that cater to
diverse preferences with 3 times less computational resources compared to
MORLHF.
