ACL Findings2025
STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework
Wenhao Liu, Zhenyi Lu, Xinyu Hu, Jerry Zhang, Dailin Li, Jiacheng Cen, Huilin Cao, Haiteng Wang, Yuhan Li, Kun Xie, Dandan Li, Pei Zhang
Abstract
High-quality math datasets are crucial for advancing the reasoning abilities of large language models (LLMs). However, existing datasets often suffer from three key issues: outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. To address these, we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers, which includes dense humanlike approximations and heuristic cues. To ensure the reliability and quality, we propose a novel human-in-the-loop, multi-agent data generation framework, integrating reasoningdense filters, multi-agent collaboration, and human mathematicians' evaluations. We curated a set of 2,000 synthetic samples and deliberately selected the 100 most difficult problems. Even most advanced models like GPT-o1 solved fewer than 5% of them. Fine-tuning on STORM-BORN boosts accuracy by 7.84% (LLaMA3-8B) and 9.12% (Qwen2.5-7B). As AI approaches mathematician-level reasoning, STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource. Our code and dataset are publicly available at https://github.com/ lwhere/STORM-BORN.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Automatic Chain of Thought Prompting in Large Language Models cites STaR: Bootstrapping Reasoning With Reasoning
- Let's Verify Step by Step cites STaR: Bootstrapping Reasoning With Reasoning
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Let's Verify Step by Step
Builds on18
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
- Automatic Chain of Thought Prompting in Large Language ModelsZhuosheng Zhang, Aston Zhang, Mu Li, Alex SmolaICLR 2023 · 234 citations
Related papers
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction DataShubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin et al.ICLR 2025
- Neuro-Symbolic Data Generation for Math ReasoningZenan Li, Zhi Zhou, Yuan Yao, Xian Zhang et al.NeurIPS 2024 · 35 citations
- FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis PipelineParker Seegmiller, Kartik Mehta, Soumya Saha, Chenyang Tao et al.EMNLP Findings 2025
- HARDMath: A Benchmark Dataset for Challenging Problems in Applied MathematicsJingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht et al.ICLR 2025
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMsSyeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh et al.ICLR 2025
