ACL Findings2026
LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation
Tianyu Liu, Qitan Lv, Hao Li, Xing Gao, Xiao Sun, Xiaoyan Sun
Abstract
Speculative decoding (SD), where a small draft model is employed to propose draft tokens in advance and then the target model validates them in parallel, has emerged as a promising technique for LLM inference acceleration. Many endeavors to improve SD are to eliminate the need for a draft model and generate draft tokens in a retrieval-based manner in order to further alleviate the drafting overhead and significantly reduce the difficulty in deployment and applications. However, retrievalbased SD relies on a matching paradigm to retrieve the most relevant reference as the draft tokens, where these methods often fail to find matched and accurate draft tokens. To address this challenge, we propose LogitSpec to effectively expand the retrieval range and find the most relevant reference as drafts. Logit-Spec is motivated by the observation that the logit of the last token can not only predict the next token, but also speculate the next next token. Specifically, LogitSpec generates draft tokens in two steps: (1) utilizing the last logit to speculate the next next token; (2) retrieving relevant reference for both the next token and the next next token. LogitSpec is training-free and plug-and-play, which can be easily integrated into existing LLM inference frameworks. Extensive experiments on a wide range of text generation benchmarks demonstrate that LogitSpec can achieve up to 2.61× speedup and 3.28 mean accepted tokens per decoding step. Our code is available at https: //github.com/smart-lty/LogitSpec.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding cites Fast Inference from Transformers via Speculative Decoding
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Fast Inference from Transformers via Speculative Decoding
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware cites Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware cites EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware cites Fast Inference from Transformers via Speculative Decoding
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism cites Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism cites EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism cites Fast Inference from Transformers via Speculative Decoding
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
- Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference cites Fast Inference from Transformers via Speculative Decoding
Cited by top-tier papers3
- Double: Breaking the Acceleration Limit via Double Retrieval Speculative ParallelismYuhao Shen, Tianyu Liu, Junyi Shen, Jinyang Wu et al.ACL 2026 · 12 citations
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and HardwareTruong Dinh Do, Nguyen-Khang Le, Le-Minh NguyenACL 2026
- Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM InferenceSiheng Xiong, Joe Zou, Faramarz Fekri, Yae Jee ChoICML 2026
Builds on24
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
- EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time TestYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangNeurIPS 2025 · 347 citations
- Break the Sequential Dependency of LLM Inference Using Lookahead DecodingYichao Fu, Peter Bailis, Ion Stoica, Hao ZhangICML 2024 · 290 citations
Related papers
- RACER: Retrieval-Augmented Contextual Rapid Speculative DecodingZihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang et al.ACL Findings 2026 · 1 citation
- DReSD: Dense Retrieval for Speculative DecodingMilan Gritta, Huiyin Xue, Gerasimos LampourasACL Findings 2025 · 1 citation
- SAM Decoding: Speculative Decoding via Suffix AutomatonYuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang et al.ACL 2025
- Speculative Decoding with CTC-based Draft Model for LLM Inference AccelerationZhuofan Wen, Shangtong Gui, Yang FengNeurIPS 2024 · 19 citations
- ConFu: Contemplate the Future for Better Speculative SamplingZongyue Qin, Raghavv Goel, Mukul Gagrani, Risheek Garrepalli et al.ICML 2026
