  Reinforcement Learning from Human Feedback (RLHF) is a widely used framework
for the training of language models. However, the process of using RLHF to
develop a language model that is well-aligned presents challenges, especially
when it comes to optimizing the reward model. Our research has found that
existing reward models, when trained using the traditional ranking objective
based on human preference data, often struggle to effectively distinguish
between responses that are more or less favorable in real-world scenarios. To
bridge this gap, our study introduces a novel method to estimate the preference
differences without the need for detailed, exhaustive labels from human
annotators. Our experimental results provide empirical evidence that
incorporating margin values into the training process significantly improves
the effectiveness of reward models. This comparative analysis not only
demonstrates the superiority of our approach in terms of reward prediction
accuracy but also highlights its effectiveness in practical applications.
