ACL Findings2024
Small Language Models Need Strong Verifiers to Self-Correct Reasoning
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
Abstract
Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs), where LLMs refine their solutions using self-generated critiques that pinpoint the errors. This work explores whether small (≤ 13B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs. We propose a novel pipeline that prompts smaller LMs to collect self-correction data that supports the training of self-refinement abilities. First, we leverage correct solutions to guide the model in critiquing their incorrect responses. Second, the generated critiques, after filtering, are used for supervised fine-tuning of the self-correcting reasoner through solution refinement. Our experimental results show improved self-correction abilities of two models on five datasets spanning math and commonsense reasoning, with notable performance gains when paired with a strong GPT-4-based verifier, though limitations are identified when using a weak self-verifier for determining when to correct. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
35 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Incentivizing LLMs to Self-Verify Their Answers cites Self-Refine: Iterative Refinement with Self-Feedback
- Incentivizing LLMs to Self-Verify Their Answers cites Let's Verify Step by Step
- Sherlock: Self-Correcting Reasoning in Vision-Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
- A Theoretical Understanding of Self-Correction through In-context Alignment cites Self-Refine: Iterative Refinement with Self-Feedback
- Let's Verify Step by Step cites Large Language Models are Zero-Shot Reasoners
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Self-Refine: Iterative Refinement with Self-Feedback
Cited by top-tier papers35
- A Theoretical Understanding of Self-Correction through In-context AlignmentYifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka et al.NeurIPS 2024 · 69 citations
- SiriuS: Self-improving Multi-agent Systems via Bootstrapped ReasoningWanjia Zhao, Mert Yüksekgönül, Shirley Wu, James Y. ZouNeurIPS 2025 · 49 citations
- Incentivizing LLMs to Self-Verify Their AnswersFuxiang Zhang, Jiacheng Xu, Chaojie Wang, Ce Cui et al.NeurIPS 2025 · 20 citations
- S^3cMath: Spontaneous Step-Level Self-Correction Makes Large Language Models Better Mathematical ReasonersYuchen Yan, Jin Jiang, Yang Liu, Yixin Cao et al.AAAI 2025 · 19 citations
- Sherlock: Self-Correcting Reasoning in Vision-Language ModelsYi Ding, Ruqi ZhangNeurIPS 2025 · 14 citations
Builds on29
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
Related papers
- Small Language Model Can Self-CorrectHaixia Han, Jiaqing Liang, Jie Shi, Qianyu He et al.AAAI 2024 · 31 citations
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations
- SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-CorrectionLing Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu et al.ICLR 2025 · 1 citation
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math ProblemsTian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-ZhuICLR 2025 · 2 citations
- CriticBench: Benchmarking LLMs for Critique-Correct ReasoningZicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo et al.ACL Findings 2024 · 16 citations
