EMNLP Findings2024
MathFish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula
Li Lucy, Tal August, Rose E. Wang, Luca Soldaini, Courtney Allison, Kyle Lo
摘要
To ensure that math curriculum is gradeappropriate and aligns with critical skills or concepts in accordance with educational standards, pedagogical experts can spend months carefully reviewing published math problems. Drawing inspiration from this process, our work presents a novel angle for evaluating language models' (LMs) mathematical abilities, by investigating whether they can discern skills and concepts enabled by math content. We contribute two datasets: one consisting of 385 fine-grained descriptions of K-12 math skills and concepts, or standards, from Achieve the Core (ATC ), and another of 9.9K math problems labeled with these standards ( MathFish). We develop two tasks for evaluating LMs' abilities to assess math problems: (1) verifying whether a problem aligns with a given standard, and (2) tagging a problem with all aligned standards. Working with experienced teachers, we find that LMs struggle to tag and verify standards linked to problems, and instead predict labels that are close to ground truth, but differ in subtle ways. We also show that LMs often generate problems that do not fully align with standards described in prompts, suggesting the need for careful scrutiny on use cases involving LMs for generating curricular materials. Finally, we categorize problems in GSM8k using math standards, allowing us to better understand why some problems are more difficult to solve for models than others.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
它引用的 Paper5
- Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant 等EMNLP Findings 2020 · 260 次引用
- Demystifying Prompts in Language Models via Perplexity EstimationHila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith 等EMNLP Findings 2023 · 109 次引用
- Math Word Problem Generation with Mathematical Consistency and Problem Context ConstraintsZichao Wang, Andrew S. Lan, Richard G. BaraniukEMNLP 2021 · 35 次引用
- Learning by Analogy: Diverse Questions Generation in Math Word ProblemZihao Zhou, Maizhen Ning, Qiufeng Wang, Jie Yao 等ACL Findings 2023 · 25 次引用
- The Unreasonable Effectiveness of Easy Training Data for Hard TasksPeter Hase, Mohit Bansal, Peter Clark, Sarah WiegreffeACL 2024 · 3 次引用
相关 Paper
- MathScale: Scaling Instruction Tuning for Mathematical ReasoningZhengyang Tang, Xingxing Zhang, Benyou Wang, Furu WeiICML 2024 · 163 次引用
- MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics BenchmarkHongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan 等ACL Findings 2024 · 16 次引用
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem SolversQintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong 等ACL 2024 · 9 次引用
- MATHWELL: Generating Educational Math Word Problems Using Teacher AnnotationsBryan R. Christ, Jonathan Kropko, Thomas HartvigsenEMNLP Findings 2024 · 6 次引用
- HARDMath: A Benchmark Dataset for Challenging Problems in Applied MathematicsJingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht 等ICLR 2025
