ACL Findings2025
DReSD: Dense Retrieval for Speculative Decoding
Milan Gritta, Huiyin Xue, Gerasimos Lampouras
Abstract
Speculative decoding (SD) accelerates Large Language Model (LLM) generation by using an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs. We focus on retrieval-based SD where the draft model retrieves the next tokens from a non-parametric datastore. Sparse retrieval (He et al., 2023, REST), which operates on the surface form of strings, is currently the dominant paradigm due to its simplicity and scalability. However, its effectiveness is limited due to the usage of short contexts and exact string matching. Instead, we introduce Dense Retrieval for Speculative Decoding (DReSD), a novel framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant token sequences for SD. Extensive experiments show that DReSD achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST).
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Language Models are Few-Shot Learners
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
- Fast Inference from Transformers via Speculative Decoding cites Language Models are Few-Shot Learners
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct cites Language Models are Few-Shot Learners
Builds on16
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionJay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar et al.NeurIPS 2024 · 727 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
Related papers
- LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token SpeculationTianyu Liu, Qitan Lv, Hao Li, Xing Gao et al.ACL Findings 2026
- RASD: Retrieval-Augmented Speculative DecodingGuofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang et al.ACL Findings 2025 · 3 citations
- SAM Decoding: Speculative Decoding via Suffix AutomatonYuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang et al.ACL 2025
- DistillSpec: Improving Speculative Decoding via Knowledge DistillationYongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon et al.ICLR 2024 · 143 citations
- RACER: Retrieval-Augmented Contextual Rapid Speculative DecodingZihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang et al.ACL Findings 2026 · 1 citation
