EMNLP Findings2024
MathFish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula
Li Lucy, Tal August, Rose E. Wang, Luca Soldaini, Courtney Allison, Kyle Lo
Abstract
To ensure that math curriculum is gradeappropriate and aligns with critical skills or concepts in accordance with educational standards, pedagogical experts can spend months carefully reviewing published math problems. Drawing inspiration from this process, our work presents a novel angle for evaluating language models' (LMs) mathematical abilities, by investigating whether they can discern skills and concepts enabled by math content. We contribute two datasets: one consisting of 385 fine-grained descriptions of K-12 math skills and concepts, or standards, from Achieve the Core (ATC ), and another of 9.9K math problems labeled with these standards ( MathFish). We develop two tasks for evaluating LMs' abilities to assess math problems: (1) verifying whether a problem aligns with a given standard, and (2) tagging a problem with all aligned standards. Working with experienced teachers, we find that LMs struggle to tag and verify standards linked to problems, and instead predict labels that are close to ground truth, but differ in subtle ways. We also show that LMs often generate problems that do not fully align with standards described in prompts, suggesting the need for careful scrutiny on use cases involving LMs for generating curricular materials. Finally, we categorize problems in GSM8k using math standards, allowing us to better understand why some problems are more difficult to solve for models than others.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant et al.EMNLP Findings 2020 · 260 citations
- Demystifying Prompts in Language Models via Perplexity EstimationHila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith et al.EMNLP Findings 2023 · 109 citations
- Math Word Problem Generation with Mathematical Consistency and Problem Context ConstraintsZichao Wang, Andrew S. Lan, Richard G. BaraniukEMNLP 2021 · 35 citations
- Learning by Analogy: Diverse Questions Generation in Math Word ProblemZihao Zhou, Maizhen Ning, Qiufeng Wang, Jie Yao et al.ACL Findings 2023 · 25 citations
- The Unreasonable Effectiveness of Easy Training Data for Hard TasksPeter Hase, Mohit Bansal, Peter Clark, Sarah WiegreffeACL 2024 · 3 citations
Related papers
- MathScale: Scaling Instruction Tuning for Mathematical ReasoningZhengyang Tang, Xingxing Zhang, Benyou Wang, Furu WeiICML 2024 · 163 citations
- MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics BenchmarkHongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan et al.ACL Findings 2024 · 16 citations
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem SolversQintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong et al.ACL 2024 · 9 citations
- MATHWELL: Generating Educational Math Word Problems Using Teacher AnnotationsBryan R. Christ, Jonathan Kropko, Thomas HartvigsenEMNLP Findings 2024 · 6 citations
- HARDMath: A Benchmark Dataset for Challenging Problems in Applied MathematicsJingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht et al.ICLR 2025
