ACL Findings2025
LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback
Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, Tianyu Liu, Baobao Chang
Abstract
In recent progress, mathematical verifiers have achieved success in mathematical reasoning tasks by validating the correctness of solutions generated by policy models. However, existing verifiers are trained with binary classification labels, which are not informative enough for the model to accurately assess the solutions. To mitigate the aforementioned insufficiency of binary labels, we introduce stepwise natural language feedback as rationale labels, that is, the correctness of each step and the detailed explanations. In this paper, we propose Math-Minos, a natural language feedback-enhanced verifier by constructing automatically generated training data and a twostage training paradigm for effective training and efficient inference. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier in both verification and reinforcement learning and also significantly alleviates the data-demanding problems of the reward model with an over 700% data efficiency improvement.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
15 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Large Language Models Cannot Self-Correct Reasoning Yet cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model cites MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model cites Let's Verify Step by Step
- Dynamic and Generalizable Process Reward Modeling cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Dynamic and Generalizable Process Reward Modeling cites Let's Verify Step by Step
- Dynamic and Generalizable Process Reward Modeling cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Dynamic and Generalizable Process Reward Modeling cites Training language models to follow instructions with human feedback
- SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning cites Let's Verify Step by Step
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Let's Verify Step by Step cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Let's Verify Step by Step cites Training language models to follow instructions with human feedback
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding cites Let's Verify Step by Step
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers15
- Unlocking Multimodal Mathematical Reasoning via Process Reward ModelRuilin Luo, Zhuofan Zheng, Lei Wang, Yifan Wang et al.NeurIPS 2025 · 38 citations
- MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought VerificationLinzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu et al.ACL 2025 · 38 citations
- Dynamic and Generalizable Process Reward ModelingZhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Qinyuan Cheng et al.ACL 2025 · 13 citations
- LaSeR: Reinforcement Learning with Last-Token Self-RewardingWenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo et al.ICLR 2026 · 12 citations
- SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward LearningYuyang Ding, Xinyu Shi, Juntao Li, Xiaobo Liang et al.NeurIPS 2025 · 10 citations
Builds on13
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations
Related papers
- SynthRL: Scaling Visual Reasoning with Verifiable Data SynthesisZijian Wu, Jinjie Ni, Xiangyan Liu, Zichen Liu et al.ACL Findings 2026
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-FoldAmrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg et al.NeurIPS 2024 · 143 citations
- Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem ProvingSara Rajaee, Kumar Pratik, Gabriele Cesa, Arash BehboodiACL Findings 2025
- What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical ReasoningYiran Ma, Zui Chen, Tianqiao Liu, Mi Tian et al.AAAI 2025 · 22 citations
- Mathesis: Towards Formal Theorem Proving from Natural LanguagesXuejun Yu, Jianyuan Zhong, Zijin Feng, Pengyi Zhai et al.ICLR 2026 · 15 citations
