EMNLP Findings2025
Stress-Testing the Reasoning Competence of Language Models With Formal Proofs
Abstract
We present a broad empirical study of stateof-the-art LLMs and LRMs (Large Reasoning Models) on PROOFGRID, a new battery of challenging but tractable logical inference tasks that form a domain-independent test of constraint-based reasoning. The tasks include proof writing and proof checking across propositional and equational logic. We also introduce two novel tasks: proof inpainting and proof gap-filling. Solving these problems requires tracking the global structure of a mathematical argument, writing hierarchical subproofs, maintaining coherence across nested assumptions, performing complex case analyses, applying inference rules, reasoning about identity and term rewriting, and reasoning about proofs themselves. Our experiments reveal impressive performance by top-tier models but also systematic failure modes. Along with the benchmarks, we release a new data resource comprising over 10K formal deduction problems and corresponding proofs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on14
- miniF2F: a cross-system benchmark for formal Olympiad-level mathematicsKunhao Zheng, Jesse Michael Han, Stanislas PoluICLR 2022 · 342 citations
- ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural LanguageOyvind Tafjord, Bhavana Dalvi, Peter ClarkACL Findings 2021 · 122 citations
- Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language ModelsHaoxiang Sun, Yingqian Min, Zhipeng Chen, Xin Zhao et al.ACL 2026 · 53 citations
- Learning Deductive Reasoning from Synthetic Corpus based on Formal LogicTerufumi Morishita, Gaku Morio, Atsuki Yamaguchi, Yasuhiro SogawaICML 2023 · 45 citations
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-ThoughtAbulhair Saparov, He HeICLR 2023 · 38 citations
Related papers
- Large Language Model for OWL ProofsHui Yang, Jiaoyan Chen, Uli SattlerWWW 2026 · 1 citation
- IsarStep: a Benchmark for High-level Mathematical ReasoningWenda Li, Lei Yu, Yuhuai Wu, Lawrence C. PaulsonICLR 2021 · 69 citations
- FormalML: A Benchmark for Evaluating Formal Subgoal Completion in Machine Learning TheoryXiao-Wen Yang, Zihao Zhang, Jianuo Cao, Zhi Zhou et al.ICLR 2026 · 8 citations
- DiLA: Enhancing LLM Tool Learning with Differential Logic LayerYu Zhang, Hui-Ling Zhen, Zehua Pei, Yingzhao Lian et al.KDD 2026 · 6 citations
- MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex ProofsAndreas Opedal, Haruki Shirakami, Bernhard Schölkopf, Abulhair Saparov et al.ICLR 2025
