Lune

EMNLP Findings2025

SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative Exploration

Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, Yong Jiang

Abstract

Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks through chain-of-thought (CoT) reasoning.However, they suffer from high inference latency due to lengthy reasoning chains. In this paper, we propose SpecCoT, a collaborative framework that combines large and small models for effective yet efficient reasoning. Unlike traditional speculative decoding, which operates at the token level, SpecCoT adopts a step-level verification strategy: the large model first establishes the reasoning direction, and for each intermediate step, the small model generates multiple candidate drafts in parallel. The large model then verifies these drafts, either selecting the most suitable one or rejecting them all and generating its own. SpecCoT approach balances reasoning quality with inference efficiency through fine-grained model cooperation. Experiments across diverse tasks show SpecCoT reduces inference latency by 1.7-4.1× while maintaining comparable accuracy to standard large model inference.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and Reduction cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Break the Sequential Dependency of LLM Inference Using Lookahead Decoding cites Fast Inference from Transformers via Speculative Decoding
  • SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification cites Fast Inference from Transformers via Speculative Decoding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and ReductionShengxuan Qiu, Haochen Huang, Shuzhang Zhong, Pengfei Zuo et al.ICML 2026 · 1 citation

Builds on15

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. Break the Sequential Dependency of LLM Inference Using Lookahead DecodingYichao Fu, Peter Bailis, Ion Stoica, Hao ZhangICML 2024 · 290 citations
  5. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and VerificationXupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng et al.ASPLOS 2024 · 105 citations

Related papers

  1. Efficient Reasoning for LLMs through Speculative Chain-of-ThoughtJikai Wang, Juntao Li, Jianye Hou, Yan Bowen et al.ACL Findings 2026
  2. SpecReason: Fast and Accurate Inference-Time Compute via Speculative ReasoningRui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro et al.NeurIPS 2025 · 68 citations
  3. ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated VerificationSiran Liu, Zane Cao, Yongchao HeACL 2026 · 3 citations
  4. Accelerating Large Language Model Reasoning via Speculative SearchZhihai Wang, Jie Wang, Jilai Pan, Xilin Xia et al.ICML 2025
  5. GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of ThoughtsWenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines