Lune

ICLR2025Top-tier venue

MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs

Andreas Opedal, Haruki Shirakami, Bernhard Schölkopf, Abulhair Saparov, Mrinmaya Sachan

Abstract

Large language models (LLMs) can solve arithmetic word problems with high accuracy, but little is known about how well they generalize to more complex problems. This is difficult to study, as (i) much of the available evaluation data has already been seen by the most capable models during training, and (ii) existing benchmarks do not capture how problem proofs may be arbitrarily complex in various ways. In this paper, we present a data-generation framework for evaluating LLMs on problems with arbitrarily complex arithmetic proofs, called MathGAP. MathGAP generates problem statements and chain-of-thought reasoning traces according to specifications about their arithmetic proof structure, enabling systematic studies on easy-to-hard generalization with respect to complexity of proof trees. Using Math-GAP, we find that LLMs show a significant decrease in performance as proofs get deeper and wider. This effect is more pronounced in complex, nonlinear proof structures, which are challenging even for the most capable models. The models are also sensitive to simple changes in sentence ordering. However, they remain capable of solving some complex problems, suggesting that reasoning generalization is noisy. https://github.com/eth-lre/mathgap-experiments

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models Can Be Easily Distracted by Irrelevant Context cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • No Need for Explanations: LLMs can implicitly learn from mistakes in-context cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Are Language Models Efficient Reasoners? A Perspective from Logic Programming cites Large Language Models Can Be Easily Distracted by Irrelevant Context
  • Are Language Models Efficient Reasoners? A Perspective from Logic Programming cites MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
  • Compositional Generalization through Gradient Search in Nonparametric Latent Space cites Faith and Fate: Limits of Transformers on Compositionality
  • Faith and Fate: Limits of Transformers on Compositionality cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Are Language Models Efficient Reasoners? A Perspective from Logic ProgrammingAndreas Opedal, Yanick Zengaffinen, Haruki Shirakami, Clemente Pasti et al.NeurIPS 2025 · 4 citations
  2. FormalScience: Scalable Human-in-the-Loop Autoformalisation of Science with Agentic Code Generation in LeanJordan Meadows, Lan Zhang, André FreitasACL 2026 · 3 citations
  3. No Need for Explanations: LLMs can implicitly learn from mistakes in-contextLisa Alazraki, Maximilian Mozes, Jon Ander Campos, Yi Chern Tan et al.EMNLP 2025
  4. Compositional Generalization through Gradient Search in Nonparametric Latent SpaceHaruki Shirakami, James HendersonICLR 2026

Builds on34

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  3. Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
  4. Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li et al.NeurIPS 2023 · 728 citations
  5. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations

Related papers

  1. MathConstruct: Challenging LLM Reasoning with Constructive ProofsMislav Balunovic, Jasper Dekoninck, Nikola Jovanovic, Ivo Petrov et al.ICML 2025
  2. From A and B to A+B: Can Large Language Models Solve Compositional Math Problems?Xisheng Xiao, Hanlin ZhaoEMNLP 2025
  3. Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical ReasoningJoykirat Singh, Akshay Uttama Nambi, Vibhav VineetACL 2025 · 10 citations
  4. GSM-∞: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian et al.ICML 2025
  5. Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided InterventionsPengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya et al.ACL Findings 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines