EMNLP Findings2024
Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation
Siru Ouyang, Shuohang Wang, Minhao Jiang, Ming Zhong, Donghan Yu, Jiawei Han, Yelong Shen
Abstract
Speculative decoding stands as a pivotal technique to expedite inference in autoregressive (large) language models. This method employs a smaller draft model to speculate a block of tokens, which the target model then evaluates for acceptance. Despite a wealth of studies aimed at increasing the efficiency of speculative decoding, the influence of generation configurations on the decoding process remains poorly understood, especially concerning decoding temperatures. This paper delves into the effects of decoding temperatures on speculative decoding's efficacy. Beginning with knowledge distillation (KD), we first highlight the challenge of decoding at higher temperatures, and demonstrate KD in a consistent temperature setting could be a remedy. We also investigate the effects of out-of-domain testing sets with out-of-range temperatures. Building upon these findings, we take an initial step to further the speedup for speculative decoding, particularly in a high-temperature generation setting. Our work offers new insights into how generation configurations drastically affect the performance of speculative decoding, and underscores the need for developing methods that focus on diverse decoding configurations. Code is publically available at https://github.com/ozyyshr/TempSpec.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- RAST: Reasoning Activation in LLMs via Small-model Transfer cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- RAST: Reasoning Activation in LLMs via Small-model Transfer cites Fast Inference from Transformers via Speculative Decoding
- RAST: Reasoning Activation in LLMs via Small-model Transfer cites The Curious Case of Neural Text Degeneration
- Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation cites Fast Inference from Transformers via Speculative Decoding
- Dynamic Speculative Agent Planning cites Fast Inference from Transformers via Speculative Decoding
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites The Curious Case of Neural Text Degeneration
- SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive Continuation cites Fast Inference from Transformers via Speculative Decoding
Cited by top-tier papers4
- Dynamic Speculative Agent PlanningYilin Guan, Qingfeng Lan, Fei Sun, Dujian Ding et al.ICLR 2026 · 10 citations
- RAST: Reasoning Activation in LLMs via Small-model TransferSiru Ouyang, Xinyu Zhu, Zilin Xiao, Minhao Jiang et al.NeurIPS 2025 · 9 citations
- SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive ContinuationJialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai et al.CVPR 2026 · 1 citation
- Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form GenerationZiyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen et al.EMNLP 2025
Builds on15
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
- EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
Related papers
- DistillSpec: Improving Speculative Decoding via Knowledge DistillationYongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon et al.ICLR 2024 · 143 citations
- Online Speculative DecodingXiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung et al.ICML 2024 · 104 citations
- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative DecodersYuezhou Hu, Jiaxin Guo, Xinyu Feng, Tuo ZhaoNeurIPS 2025 · 9 citations
- A Theoretical Perspective for Speculative Decoding AlgorithmMing Yin, Minshuo Chen, Kaixuan Huang, Mengdi WangNeurIPS 2024 · 36 citations
- Block Verification Accelerates Speculative DecodingZiteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni et al.ICLR 2025
