Lune

ACL Findings2025

Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning

Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, Anh Tuan Luu

Abstract

Direct Preference Optimization (DPO) often struggles with long-chain mathematical reasoning. Existing approaches, such as Step-DPO, typically improve this by focusing on the first erroneous step in the reasoning chain. However, they overlook all other steps and rely heavily on humans or GPT-4 to identify erroneous steps. To address these issues, we propose Full-Step-DPO, a novel DPO framework tailored for mathematical reasoning. Instead of optimizing only the first erroneous step, it leverages step-wise rewards from the entire reasoning chain. This is achieved by training a self-supervised process reward model, which automatically scores each step, providing rewards while avoiding reliance on external signals. Furthermore, we introduce a novel stepwise DPO loss, which dynamically updates gradients based on these step-wise rewards. This endows stronger reasoning capabilities to language models. Extensive evaluations on both in-domain and out-of-domain mathematical reasoning benchmarks across various base language models, demonstrate that Full-Step-DPO achieves superior performance compared to state-of-the-art baselines 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language Models cites Solving Quantitative Reasoning Problems with Language Models
  • Let's Verify Step by Step cites Solving Quantitative Reasoning Problems with Language Models
  • Learning Rewrite-Invariant Reasoning with Targeted Alternation Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM ReasoningZhihao Dou, Qinjian Zhao, Zhongwei Wan, Zhang Dinggen et al.ICML 2026 · 24 citations
  2. Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language ModelsJinman Zhao, Erxue Min, Hui Wu, Ziheng Li et al.AAAI 2026 · 1 citation
  3. Learning Rewrite-Invariant Reasoning with Targeted Alternation TrainingMousa Arraf, Ido Guy, Kira RadinskyICML 2026

Builds on23

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  3. Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
  4. Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
  5. Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations

Related papers

  1. TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference TreesWeibin Liao, Xu Chu, Yasha WangICLR 2025
  2. Step-level Value Preference Optimization for Mathematical ReasoningGuoxin Chen, Minpeng Liao, Chengxi Li, Kai FanEMNLP Findings 2024 · 6 citations
  3. Process-based Self-Rewarding Language ModelsShimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu et al.ACL Findings 2025 · 1 citation
  4. Learning Planning-based Reasoning by Trajectories Collection and Process Reward SynthesizingFangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen et al.EMNLP 2024 · 1 citation
  5. Self-Training with Direct Preference Optimization Improves Chain-of-Thought ReasoningTianduo Wang, Shichen Li, Wei LuACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines