Lune

ACL Findings2026

FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning

Xiao Li, Bolin Zhu, Kaiwen Shi, Sichen Liu, Yin Zhu, Yiwei Liu, Gong Cheng

Abstract

The application of physics formulas is a fundamental human capability in numerical reasoning.While existing datasets often rely on implicit mathematical knowledge, they rarely explicitate the underlying formulas.To address this, we introduce FormulaReasoning, a new benchmark for formula-based numerical reasoning comprising 5,324 questions requiring calculations grounded in external physics principles.We provide high-quality, fine-grained annotations in English and Chinese-including formula structures, parameter names, symbols, values, and units-curated through manual effort and LLM-assisted validation.Additionally, we provide a consolidated formula database as an external knowledge source.To further challenge model performance, we develop an extended version of the dataset by coupling multiple questions.We evaluate various architectural and methodological frameworks, including retrieval-augmented methods, modular reasoning (formula generation, parameter extraction, and calculation), and preferencebased optimization.Our analysis identifies critical challenges in formula-based reasoning, highlighting significant opportunities for future methodological advancement.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Large Language Models are Zero-Shot Reasoners
  • Large Language Models are Zero-Shot Reasoners cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
  4. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  5. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations

Related papers

  1. Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem SolvingShunfeng Zheng, Yudi Zhang, Meng Fang, Zihan Zhang et al.EMNLP Findings 2025 · 1 citation
  2. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language ModelsXin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen et al.ICML 2025
  3. PhysReason: A Comprehensive Benchmark towards Physics-Based ReasoningXinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang et al.ACL 2025 · 51 citations
  4. PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution DimensionsSong Dai, Yibo Yan, Jiamin Su, Dongfang Zihao et al.EMNLP Findings 2025 · 2 citations
  5. FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step ComputationZichen Tang, Haihong E, Rongjin Li, Jiacheng Liu et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines