ACL Findings2025
Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning
Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, Anh Tuan Luu
Abstract
Direct Preference Optimization (DPO) often struggles with long-chain mathematical reasoning. Existing approaches, such as Step-DPO, typically improve this by focusing on the first erroneous step in the reasoning chain. However, they overlook all other steps and rely heavily on humans or GPT-4 to identify erroneous steps. To address these issues, we propose Full-Step-DPO, a novel DPO framework tailored for mathematical reasoning. Instead of optimizing only the first erroneous step, it leverages step-wise rewards from the entire reasoning chain. This is achieved by training a self-supervised process reward model, which automatically scores each step, providing rewards while avoiding reliance on external signals. Furthermore, we introduce a novel stepwise DPO loss, which dynamically updates gradients based on these step-wise rewards. This endows stronger reasoning capabilities to language models. Extensive evaluations on both in-domain and out-of-domain mathematical reasoning benchmarks across various base language models, demonstrate that Full-Step-DPO achieves superior performance compared to state-of-the-art baselines 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language Models cites Solving Quantitative Reasoning Problems with Language Models
- Let's Verify Step by Step cites Solving Quantitative Reasoning Problems with Language Models
- Learning Rewrite-Invariant Reasoning with Targeted Alternation Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Cited by top-tier papers3
- Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM ReasoningZhihao Dou, Qinjian Zhao, Zhongwei Wan, Zhang Dinggen et al.ICML 2026 · 24 citations
- Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language ModelsJinman Zhao, Erxue Min, Hui Wu, Ziheng Li et al.AAAI 2026 · 1 citation
- Learning Rewrite-Invariant Reasoning with Targeted Alternation TrainingMousa Arraf, Ido Guy, Kira RadinskyICML 2026
Builds on23
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
Related papers
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference TreesWeibin Liao, Xu Chu, Yasha WangICLR 2025
- Step-level Value Preference Optimization for Mathematical ReasoningGuoxin Chen, Minpeng Liao, Chengxi Li, Kai FanEMNLP Findings 2024 · 6 citations
- Process-based Self-Rewarding Language ModelsShimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu et al.ACL Findings 2025 · 1 citation
- Learning Planning-based Reasoning by Trajectories Collection and Process Reward SynthesizingFangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen et al.EMNLP 2024 · 1 citation
- Self-Training with Direct Preference Optimization Improves Chain-of-Thought ReasoningTianduo Wang, Shichen Li, Wei LuACL 2024
