Reasoning Fails Where Step Flow Breaks
Xiaoyu Xu, Yulan Pan, Xiaosong Yuan, Zhihong Shen, Minghao Su, Yuanhao Su, Xiaofeng Zhang
摘要
Large reasoning models (LRMs) that generate long chains of thought now perform well on multi-step math, science, and coding tasks. However, their behavior is still unstable and hard to interpret, and existing analysis tools struggle with such long, structured reasoning traces. We introduce Step-Saliency, which pools attention-gradient scores into step-to-step maps along the questionthinking-summary trajectory. Across several models, Step-Saliency reveals two recurring information-flow failures: Shallow Lock-in, where shallow layers over-focus on the current step and barely use earlier context, and Deep Decay, where deep layers gradually lose saliency on the thinking segment and the summary increasingly attends to itself and the last few steps. Motivated by these patterns, we propose StepFlow, a saliency-inspired test-time intervention that adjusts shallow saliency patterns measured by Step-Saliency via Odds-Equal Bridge and adds a small step-level residual in deep layers via Step Momentum Injection. StepFlow improves accuracy on math, science, and coding tasks across multiple LRMs without retraining, indicating that repairing information flow can recover part of their missing reasoning performance. Code is available at https://github. com/XiaoyuXu-Vincent/step-saliency .
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper14
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo 等NeurIPS 2022 · 被引用 8,168 次
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards 等ICLR 2024 · 被引用 3,045 次
- Attention is Not Only a Weight: Analyzing Transformers with Vector NormsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2020 · 被引用 138 次
- AttentionViz: A Global View of Transformer AttentionCatherine Yeh, Yida Chen, Aoyu Wu, Cynthia Chen 等IEEE VIS 2023 · 被引用 78 次
- Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMsQingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu 等ICLR 2024 · 被引用 76 次
相关 Paper
- What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoTYunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain 等ICML 2026
- RFS-Guard: Detecting Reasoning Hallucinations via Cross-Phase Routing Focus in Large Reasoning ModelsZihang Liu, Zhouhua Fang, Hui Liu, Zhiwei Liu 等ACL 2026
- Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language ModelsZeping Yu, Yonatan Belinkov, Sophia AnaniadouEMNLP 2025 · 被引用 2 次
- From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 ModelsJue Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei ZhangEMNLP 2025
- Characterizing and Mitigating Reasoning Drift in Large Language ModelsYufeng Zhang, Xuepeng Wang, Lingxiang Wu, Jinqiao WangICLR 2026
