Lune

ACL Findings2026

Efficient Reasoning for LLMs through Speculative Chain-of-Thought

Jikai Wang, Juntao Li, Jianye Hou, Yan Bowen, Lijun Wu, Min Zhang

Abstract

Large reasoning language models such as OpenAI-o1 and Deepseek-R1 have recently attracted widespread attention due to their impressive task-solving abilities. However, the enormous model size and the generation of lengthy thought chains introduce significant reasoning costs and response latency. Existing methods for efficient reasoning mainly focus on reducing the number of model parameters or shortening the chain-of-thought length. In this paper, we introduce Speculative Chainof-Thought (SCoT), which reduces reasoning latency from another perspective by accelerated average reasoning speed through large and small model collaboration. SCoT conducts thought-level drafting using a lightweight draft model. Then it selects the best CoT draft and corrects the error cases with the target model. The proposed thinking behavior alignment improves the efficiency of drafting and the draft selection strategy maintains the prediction accuracy of the target model for complex tasks. Experimental results on GSM8K, MATH, GaoKao, CollegeMath and Olympiad datasets show that SCoT reduces reasoning latency by 48%∼66% and 21%∼49% for Deepseek-R1-Distill-Qwen-32B and Deepseek-R1-Distill-Llama-70B while achieving neartarget-model-level performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Speculative Actions: A Lossless Framework for Faster AI Agents cites Fast Inference from Transformers via Speculative Decoding
  • Time-Frequency Token Advantage Clipping for Training Efficient Large Reasoning Model cites Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL
  • Time-Frequency Token Advantage Clipping for Training Efficient Large Reasoning Model cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLSongjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian et al.NeurIPS 2025 · 69 citations
  2. TrimR: Verifier-based Training-Free Thinking Trimming for Efficient Test-Time ScalingWeizhe Lin, Xing Li 023, Zhiyuan Yang, Xiaojin Fu et al.ICLR 2026 · 14 citations
  3. Speculative Actions: A Lossless Framework for Faster AI AgentsNaimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu et al.ICLR 2026 · 9 citations
  4. Time-Frequency Token Advantage Clipping for Training Efficient Large Reasoning ModelRong Bao, Bo Wang, Xiao Wang, Hongyu Li et al.AAAI 2026

Builds on18

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
  4. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
  5. EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations

Related papers

  1. SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative ExplorationJunhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao et al.EMNLP Findings 2025 · 2 citations
  2. Accelerating Large Language Model Reasoning via Speculative SearchZhihai Wang, Jie Wang, Jilai Pan, Xilin Xia et al.ICML 2025
  3. Latent-Guided Reasoning: Empowering Small LLMs with Large-Model ThinkingHanzhu Chen, Lin Yang, Jie Wang, Junhao Yan et al.ICLR 2026
  4. SpecReason: Fast and Accurate Inference-Time Compute via Speculative ReasoningRui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro et al.NeurIPS 2025 · 68 citations
  5. VeriThinker: Learning to Verify Makes Reasoning Model EfficientZigeng Chen, Xinyin Ma, Gongfan Fang, Ruonan Yu et al.NeurIPS 2025 · 30 citations
Dusk over the sea between two cliffs drawn in fine vertical lines