Lune

ACL Findings2026

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

Zetian Ouyang, Linlin Wang, Gerard de Melo, Liang He

摘要

Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks.We introduce PyraMath-Bench 1 , a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities.Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions.To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection).Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training. 1 https://github.com/optifine233-ship-it/Key Units: 30 (meters^2) 22 (meters) To solve this MWP, I have to 1. Abstracting problems.2. extract key numbers and units, 3. solve a system of equations, … To solve this VRP, I have to 1. Abstracting problems.2. extract key numbers, 3. extract information from the picture, … L=6, W=5 or L=5, W=6 And L > W, so L=6 meters, W=5 meters The question can be abstracted as Unknown Variables: L,W Equations: 2*(L+W)=22, L*W=30 The question can be abstracted as 15 + 15 + 30 = ?60 / 2 = ?-(-11) ± (-11) 2 -4 * 30 2 = 𝟔, 𝟓 15 + 15 + 30 = 60 60 / 2 = 30 How many clocks are there in the picture? 3 Minutes of the digital clock: 15 Minutes of the analog clock 15, 30

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Gorilla: Large Language Model Connected with Massive APIs 引用了 Toolformer: Language Models Can Teach Themselves to Use Tools
  • Gorilla: Large Language Model Connected with Massive APIs 引用了 HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs 引用了 Gorilla: Large Language Model Connected with Massive APIs
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs 引用了 Toolformer: Language Models Can Teach Themselves to Use Tools
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs 引用了 HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face 引用了 Toolformer: Language Models Can Teach Themselves to Use Tools
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper14

  1. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu 等NeurIPS 2023 · 5,989 次引用
  2. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li 等NeurIPS 2023 · 1,778 次引用
  3. Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 次引用
  4. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu 等ICLR 2024 · 1,472 次引用
  5. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu 等ICLR 2024 · 1,469 次引用

相关 Paper

  1. Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language ModelsHaoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li 等ACL Findings 2025 · 12 次引用
  2. Number Cookbook: Number Understanding of Language Models and How to Improve ItHaotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin 等ICLR 2025
  3. DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Financial DocumentsYilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi 等ACL 2024 · 8 次引用
  4. MathScale: Scaling Instruction Tuning for Mathematical ReasoningZhengyang Tang, Xingxing Zhang, Benyou Wang, Furu WeiICML 2024 · 163 次引用
  5. Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical ReasoningJoykirat Singh, Akshay Uttama Nambi, Vibhav VineetACL 2025 · 10 次引用
黄昏的海面,两侧是细线勾勒的悬崖