Lune

EMNLP Findings2024

Step-level Value Preference Optimization for Mathematical Reasoning

Guoxin Chen, Minpeng Liao, Chengxi Li, Kai Fan

Abstract

Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the finegrained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-ofdomain mathematical reasoning benchmarks. Our code is available at https://github. com/MARIO-Math-Reasoning/Super_MARIO.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

31 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards cites What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning
  • Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization cites Solving Quantitative Reasoning Problems with Language Models
  • DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization cites Training language models to follow instructions with human feedback
  • RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers31

  1. RL Tango: Reinforcing Generator and Verifier Together for Language ReasoningKaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wei Hong et al.NeurIPS 2025 · 44 citations
  2. Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table ReasoningPeiying Yu, Guoxin Chen, Jingjing WangACL 2025 · 30 citations
  3. Optimas: Optimizing Compound AI Systems with Globally Aligned Local RewardsShirley Wu, Parth Sarthi, Shiyu Zhao, Aaron Lee et al.ICLR 2026 · 25 citations
  4. DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained OptimizationGang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu et al.NeurIPS 2025 · 24 citations
  5. What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical ReasoningYiran Ma, Zui Chen, Tianqiao Liu, Mi Tian et al.AAAI 2025 · 22 citations

Builds on24

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
  4. Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
  5. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations

Related papers

  1. TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference TreesWeibin Liao, Xu Chu, Yasha WangICLR 2025
  2. Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined ValuesHongbo Zhang, Han Cui, Guangsheng Bao, Linyi Yang et al.EMNLP 2025 · 6 citations
  3. Language Models as Implicit Tree SearchZiliang Chen, Zhao-Rong Lai, Yufeng Yang, Liangda Fang et al.ICML 2025
  4. Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language ModelsJinman Zhao, Erxue Min, Hui Wu, Ziheng Li et al.AAAI 2026 · 1 citation
  5. Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
Dusk over the sea between two cliffs drawn in fine vertical lines