EMNLP Findings2025
SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative Exploration
Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, Yong Jiang
Abstract
Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks through chain-of-thought (CoT) reasoning.However, they suffer from high inference latency due to lengthy reasoning chains. In this paper, we propose SpecCoT, a collaborative framework that combines large and small models for effective yet efficient reasoning. Unlike traditional speculative decoding, which operates at the token level, SpecCoT adopts a step-level verification strategy: the large model first establishes the reasoning direction, and for each intermediate step, the small model generates multiple candidate drafts in parallel. The large model then verifies these drafts, either selecting the most suitable one or rejecting them all and generating its own. SpecCoT approach balances reasoning quality with inference efficiency through fine-grained model cooperation. Experiments across diverse tasks show SpecCoT reduces inference latency by 1.7-4.1× while maintaining comparable accuracy to standard large model inference.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and Reduction cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding cites Fast Inference from Transformers via Speculative Decoding
- SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification cites Fast Inference from Transformers via Speculative Decoding
Cited by top-tier papers1
Builds on15
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- Break the Sequential Dependency of LLM Inference Using Lookahead DecodingYichao Fu, Peter Bailis, Ion Stoica, Hao ZhangICML 2024 · 290 citations
- SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and VerificationXupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng et al.ASPLOS 2024 · 105 citations
Related papers
- Efficient Reasoning for LLMs through Speculative Chain-of-ThoughtJikai Wang, Juntao Li, Jianye Hou, Yan Bowen et al.ACL Findings 2026
- SpecReason: Fast and Accurate Inference-Time Compute via Speculative ReasoningRui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro et al.NeurIPS 2025 · 68 citations
- ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated VerificationSiran Liu, Zane Cao, Yongchao HeACL 2026 · 3 citations
- Accelerating Large Language Model Reasoning via Speculative SearchZhihai Wang, Jie Wang, Jilai Pan, Xilin Xia et al.ICML 2025
- GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of ThoughtsWenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu et al.ACL Findings 2026
