Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
Valentin NOËL
摘要
Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle. We show that valid mathematical reasoning induces a measurable, training-free spectral signature in transformer attention. By treating each attention matrix as a weighted token graph, we extract four diagnostics: Fiedler value, High-Frequency Energy Ratio (HFER), spectral entropy, and smoothness, that require no learned parameters. Experiments across seven models from four architectural families yield effect sizes up to Cohen's (), enabling -% single-threshold classification accuracy. Two findings sharpen the interpretation. First, Platonic validity: the spectral signal tracks logical coherence rather than compiler acceptance, proofs rejected for timeouts or missing imports are correctly classified as valid, a distinction confirmed by a manual audit (, ). Second, architectural determinism: Sliding Window Attention shifts the discriminative feature from HFER to smoothness (, ), showing that attention design governs which spectral channel encodes reasoning quality. Causal ablation confirms the signature traces induction-head circuits. The method generalises to informal chain-of-thought (, ), and in proof search, HFER reranking improves Best-of-16 Pass@1 by -%, matching % of the AUC of fully supervised probes with zero labels. Spectral graph analysis is a principled, architecture-aware primitive for reasoning verification.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper21
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 被引用 3,415 次
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards 等ICLR 2024 · 被引用 3,045 次
- Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer 等NeurIPS 2022 · 被引用 2,039 次
- Rethinking Graph Transformers with Spectral AttentionDevin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau 等NeurIPS 2021 · 被引用 854 次
- Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li 等NeurIPS 2023 · 被引用 728 次
相关 Paper
- Verifying Chain-of-Thought Reasoning via Its Computational GraphZheng Zhao, Yeskendir Koishekenov, Xianjun Yang, Naila Murray 等ICLR 2026 · 被引用 25 次
- HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMsAzim Ospanov, Zijin Feng, Jiacheng Sun, Haoli Bai 等ICML 2026 · 被引用 5 次
- RFS-Guard: Detecting Reasoning Hallucinations via Cross-Phase Routing Focus in Large Reasoning ModelsZihang Liu, Zhouhua Fang, Hui Liu, Zhiwei Liu 等ACL 2026
- Reasoning Structure of Large Language ModelsFrédéric Berdoz, Luca Lanzendörfer, Fabian Farestam, Roger WattenhoferICML 2026
- Who Reasons in the Large Language Models?Jie Shao, Jianxin WuNeurIPS 2025 · 被引用 7 次
