EMNLP Findings2024
Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, Jingbo Shang
Abstract
Process supervision, using a trained verifier to evaluate the intermediate steps generated by a reasoner, has demonstrated significant improvements in multi-step problem solving. In this paper, to avoid the expensive effort of human annotation on the verifier training data, we introduce Model-induced Process Supervision (MiPS), a novel method for automating data curation. MiPS annotates an intermediate step by sampling completions of this solution through the reasoning model, and obtaining an accuracy defined as the proportion of correct completions. Inaccuracies of the reasoner would cause MiPS underestimating the accuracy of intermediate steps, therefore, we suggest and empirically show that verification focusing on high predicted scores of the verifier shall be preferred over that of low predicted scores, contrary to prior observations on human curated data. Our approach significantly improves the performance of PaLM 2 on math and coding tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with an output supervision trained verifier). Additionally, our study demonstrates that the verifier exhibits strong generalization ability across different reasoning models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
27 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision cites Let's Verify Step by Step
- Large Language Models Cannot Self-Correct Reasoning Yet cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Large Language Models Cannot Self-Correct Reasoning Yet cites Self-Refine: Iterative Refinement with Self-Feedback
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
- DreamPRM: Domain-reweighted Process Reward Model for Multimodal Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- DreamPRM: Domain-reweighted Process Reward Model for Multimodal Reasoning cites Let's Verify Step by Step
- Let's Verify Step by Step cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- AutoPSV: Automated Process-Supervised Verifier cites Large Language Models Cannot Self-Correct Reasoning Yet
- AutoPSV: Automated Process-Supervised Verifier cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- AutoPSV: Automated Process-Supervised Verifier cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- AutoPSV: Automated Process-Supervised Verifier cites Self-Refine: Iterative Refinement with Self-Feedback
- AutoPSV: Automated Process-Supervised Verifier cites Let's Verify Step by Step
- Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Cited by top-tier papers27
- Easy-to-Hard Generalization: Scalable Alignment Beyond Human SupervisionZhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu et al.NeurIPS 2024 · 125 citations
- Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM ReasoningMaggie Ziyu Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu et al.ICML 2026 · 102 citations
- AutoPSV: Automated Process-Supervised VerifierJianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao et al.NeurIPS 2024 · 35 citations
- DreamPRM: Domain-reweighted Process Reward Model for Multimodal ReasoningQi Cao, Ruiyi Wang, Ruiyi Zhang, Sai Ashish Somayajula et al.NeurIPS 2025 · 17 citations
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time ScalingXinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan et al.NeurIPS 2025 · 16 citations
Builds on13
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
Related papers
- Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human AnnotationsPeiyi Wang, Lei Li, Zhihong Shao, Runxin Xu et al.ACL 2024
- SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward ModellingMd Imbesat Hassan Rizvi, Xiaodan Zhu, Iryna GurevychAAAI 2026 · 2 citations
- Reasoning Through Execution: Unifying Process and Outcome Rewards for Code GenerationZhuohao Yu, Weizheng Gu, Yidong Wang, Xingru Jiang et al.ICML 2025
- LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language FeedbackBofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang et al.ACL Findings 2025 · 3 citations
- Efficient PRM Training Data Synthesis via Formal VerificationRyo Kamoi, Yusen Zhang, Nan Zhang, Sarkar Snigdha Sarathi Das et al.ACL Findings 2026
