ACL Findings2026
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He
Abstract
Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks.We introduce PyraMath-Bench 1 , a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities.Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions.To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection).Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training. 1 https://github.com/optifine233-ship-it/Key Units: 30 (meters^2) 22 (meters) To solve this MWP, I have to 1. Abstracting problems.2. extract key numbers and units, 3. solve a system of equations, … To solve this VRP, I have to 1. Abstracting problems.2. extract key numbers, 3. extract information from the picture, … L=6, W=5 or L=5, W=6 And L > W, so L=6 meters, W=5 meters The question can be abstracted as Unknown Variables: L,W Equations: 2*(L+W)=22, L*W=30 The question can be abstracted as 15 + 15 + 30 = ?60 / 2 = ?-(-11) ± (-11) 2 -4 * 30 2 = 𝟔, 𝟓 15 + 15 + 30 = 60 60 / 2 = 30 How many clocks are there in the picture? 3 Minutes of the digital clock: 15 Minutes of the analog clock 15, 30
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Gorilla: Large Language Model Connected with Massive APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Gorilla: Large Language Model Connected with Massive APIs cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs cites Gorilla: Large Language Model Connected with Massive APIs
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Toolformer: Language Models Can Teach Themselves to Use Tools
Builds on14
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations
- Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu et al.ICLR 2024 · 1,469 citations
Related papers
- Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language ModelsHaoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li et al.ACL Findings 2025 · 12 citations
- Number Cookbook: Number Understanding of Language Models and How to Improve ItHaotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin et al.ICLR 2025
- DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Financial DocumentsYilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi et al.ACL 2024 · 8 citations
- MathScale: Scaling Instruction Tuning for Mathematical ReasoningZhengyang Tang, Xingxing Zhang, Benyou Wang, Furu WeiICML 2024 · 163 citations
- Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical ReasoningJoykirat Singh, Akshay Uttama Nambi, Vibhav VineetACL 2025 · 10 citations
