Lune

ACL Findings2025

STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework

Wenhao Liu, Zhenyi Lu, Xinyu Hu, Jerry Zhang, Dailin Li, Jiacheng Cen, Huilin Cao, Haiteng Wang, Yuhan Li, Kun Xie, Dandan Li, Pei Zhang

Abstract

High-quality math datasets are crucial for advancing the reasoning abilities of large language models (LLMs). However, existing datasets often suffer from three key issues: outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. To address these, we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers, which includes dense humanlike approximations and heuristic cues. To ensure the reliability and quality, we propose a novel human-in-the-loop, multi-agent data generation framework, integrating reasoningdense filters, multi-agent collaboration, and human mathematicians' evaluations. We curated a set of 2,000 synthetic samples and deliberately selected the 100 most difficult problems. Even most advanced models like GPT-o1 solved fewer than 5% of them. Fine-tuning on STORM-BORN boosts accuracy by 7.84% (LLaMA3-8B) and 9.12% (Qwen2.5-7B). As AI approaches mathematician-level reasoning, STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource. Our code and dataset are publicly available at https://github.com/ lwhere/STORM-BORN.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Automatic Chain of Thought Prompting in Large Language Models cites STaR: Bootstrapping Reasoning With Reasoning
  • Let's Verify Step by Step cites STaR: Bootstrapping Reasoning With Reasoning
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Let's Verify Step by Step
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on18

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  3. STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
  4. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
  5. Automatic Chain of Thought Prompting in Large Language ModelsZhuosheng Zhang, Aston Zhang, Mu Li, Alex SmolaICLR 2023 · 234 citations

Related papers

  1. OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction DataShubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin et al.ICLR 2025
  2. Neuro-Symbolic Data Generation for Math ReasoningZenan Li, Zhi Zhou, Yuan Yao, Xian Zhang et al.NeurIPS 2024 · 35 citations
  3. FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis PipelineParker Seegmiller, Kartik Mehta, Soumya Saha, Chenyang Tao et al.EMNLP Findings 2025
  4. HARDMath: A Benchmark Dataset for Challenging Problems in Applied MathematicsJingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht et al.ICLR 2025
  5. MIND: Math Informed syNthetic Dialogues for Pretraining LLMsSyeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines