  Direct Preference Optimization (DPO) using an implicit reward model has
proven to be an effective alternative to reinforcement learning from human
feedback (RLHF) for fine-tuning preference aligned large language models
(LLMs). However, the overall preference annotations of responses do not fully
capture the fine-grained quality of model outputs in complex multi-step
reasoning tasks, such as mathematical reasoning. To address this limitation, we
introduce a novel algorithm called Step-level Value Preference Optimization
(SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically
annotate step-level preferences for multi-step reasoning. Furthermore, from the
perspective of learning-to-rank, we train an explicit value model to replicate
the behavior of the implicit reward model, complementing standard preference
optimization. This value model enables the LLM to generate higher reward
responses with minimal cost during inference. Experimental results demonstrate
that our method achieves state-of-the-art performance on both in-domain and
out-of-domain mathematical reasoning benchmarks.
