Lune

ACL Findings2024

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, Fuli Feng

Abstract

The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions.Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction.From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps.We also design diverse prompts to thoroughly evaluate eleven representative LLMs.Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closedsource models GPT-3.5 and Gemini Pro.Notably, calculation error proves the most challenging error type.Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9%.These results reveal potential directions for developing the mathematical reasoning abilities of LLMs.Our code and dataset is available on https://github.com/LittleCirc1e/EIC.Question: James buys 6 t-shirts for 50% off.They each cost 20.Howmuchdidhepay?CorrectSolution:Step1:Eachshirtcost20∗.5=20.How much did he pay?Correct Solution: Step1: Each shirt cost 20*.5=10with the sale Step 2: So he paid 106=$60 Correct Answer: 60 Wrong Solution: Step1: Each shirt cost 20.5=

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • ReActR: Reasoning through Error-Activated Reflection for LLM Post-Training cites MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
  • Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models cites Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
  • Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Can Vision-Language Models Evaluate Handwritten Math?Oikantik Nath, Hanani Bathina, Mohammed Safi Ur Rahman Khan, Mitesh M. KhapraACL 2025 · 10 citations
  2. PRISM-Physics: Causal DAG-Based Process Evaluation for Physics ReasoningWanjia Zhao, Qinwei Ma, Jingzhe Shi, Shirley Wu et al.ICLR 2026 · 4 citations
  3. Improving LLM-based Log Parsing by Learning from Errors in Reasoning TracesJialai Wang, Juncheng Lu, Jie Yang, Junjie Wang et al.ASE 2025 · 2 citations
  4. Causal-R: A Causal-Reasoning Geometry Problem Solver for Optimized Solution ExplorationWenjun Wu, Lingling Zhang, Bo Zhao, Muye Huang et al.NeurIPS 2025 · 2 citations
  5. ReActR: Reasoning through Error-Activated Reflection for LLM Post-TrainingLina SunACL 2026

Builds on10

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
  3. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve ThemMirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann et al.ACL Findings 2023 · 253 citations
  4. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language ModelsLei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu et al.ACL 2023 · 249 citations
  5. Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-VerificationAojun Zhou, Ke Wang, Zimu Lu, Weikang Shi et al.ICLR 2024 · 206 citations

Related papers

  1. Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical ReasoningJoykirat Singh, Akshay Uttama Nambi, Vibhav VineetACL 2025 · 10 citations
  2. Conditional and Modal Reasoning in Large Language ModelsWesley H. Holliday, Matthew Mandelkern, Cedegao ZhangEMNLP 2024 · 5 citations
  3. ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error DetectionYibo Yan, Shen Wang, Jiahao Huo, Hang Li et al.ACL Findings 2026 · 1 citation
  4. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language ModelsDaman Arora, Himanshu Gaurav Singh, MausamEMNLP 2023 · 36 citations
  5. Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive FrameworkZhangyue Yin, Yuhong Sun, Xuanjing Huang, Xipeng Qiu et al.EMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines