GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar
Abstract
Recent advancements in Large Language Models (LLMs) have sparked interest in their mathematical reasoning capabilities. While performance on the widely popular GSM8K benchmark has improved, questions remain about whether reported evaluation metrics are reliable, and reasoning abilities of LLMs have advanced. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models. Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data. When we add a single clause that appears relevant to the question, we observe significant performance drops (up to 65%) across all state-of-the-art models, even though the added clause does not contribute to the reasoning chain needed to reach the final answer. Overall, our work provides a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
110 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Large Language Models Can Be Easily Distracted by Irrelevant Context cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- What Algorithms can Transformers Learn? A Study in Length Generalization cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain of Thought Empowers Transformers to Solve Inherently Serial Problems cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers110
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem ComplexityParshin Shojaee, Iman Mirzadeh, Keivan Alizadeh-Vahid, Maxwell Horton et al.NeurIPS 2025 · 507 citations
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language ModelsCharlie Zhang, Graham Neubig, Xiang YueICML 2026 · 58 citations
- Hyperbolic Fine-Tuning for Large Language ModelsMenglin Yang, Ram Samarth B. B., Aosong Feng, Bo Xiong et al.NeurIPS 2025 · 31 citations
- Mechanistic Detection and Mitigation of Hallucination in Large Reasoning ModelsZhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang et al.ICLR 2026 · 30 citations
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model EvaluationDavid Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu et al.NeurIPS 2025 · 24 citations
Builds on15
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
- Are Emergent Abilities of Large Language Models a Mirage?Rylan Schaeffer, Brando Miranda, Sanmi KoyejoNeurIPS 2023 · 796 citations
- Chain of Thought Empowers Transformers to Solve Inherently Serial ProblemsZhiyuan Liu, Hong Liu, Denny Zhou, Tengyu MaICLR 2024 · 259 citations
- What Algorithms can Transformers Learn? A Study in Length GeneralizationHattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin et al.ICLR 2024 · 189 citations
Related papers
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem SolversQintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong et al.ACL 2024 · 9 citations
- How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled BenchmarkMinglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu et al.EMNLP 2025 · 2 citations
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning ProcessTian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-ZhuICLR 2025 · 3 citations
- Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided InterventionsPengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya et al.ACL Findings 2025 · 5 citations
- MARIO: MAth Reasoning with code Interpreter Output - A Reproducible PipelineMinpeng Liao, Chengxi Li, Wei Luo, Jing Wu et al.ACL Findings 2024 · 2 citations
