ACL Findings2025
RASD: Retrieval-Augmented Speculative Decoding
Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, Hao Henry Wang
Abstract
Speculative decoding accelerates inference in large language models (LLMs) by generating draft tokens for target model verification. Current approaches for obtaining draft tokens rely on lightweight draft models or additional model structures to generate draft tokens and retrieve context from databases. Due to the draft model's small size and limited training data, model-based speculative decoding frequently becomes less effective in out-of-domain scenarios. Additionally, the time cost of the drafting phase results in a low upper limit on acceptance length during the verification step, limiting overall efficiency. This paper proposes RASD (Retrieval-Augmented Speculative Decoding), which adopts retrieval methods to enhance model-based speculative decoding. We introduce tree pruning and tree fusion to achieve this. Specifically, we develop a pruning method based on the draft model's probability distribution to construct the optimal retrieval tree. Second, we employ the longest prefix matching algorithm to merge the tree generated by the draft model with the retrieval tree, resulting in a unified tree for verification. Experimental results demonstrate that RASD achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA. Moreover, RASD exhibits strong scalability, seamlessly integrating with various speculative decoding approaches, including both generation-based and retrievalbased methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Talon: Breaking the Synchronization Barrier in Speculative Decoding with Hybrid Model-based and Retrieve-based Drafting cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Talon: Breaking the Synchronization Barrier in Speculative Decoding with Hybrid Model-based and Retrieve-based Drafting cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Talon: Breaking the Synchronization Barrier in Speculative Decoding with Hybrid Model-based and Retrieve-based Drafting cites Fast Inference from Transformers via Speculative Decoding
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Language Models are Few-Shot Learners
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
- Fast Inference from Transformers via Speculative Decoding cites Language Models are Few-Shot Learners
- OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding cites Fast Inference from Transformers via Speculative Decoding
Cited by top-tier papers3
- OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decodingRamchalam Kinattinkara Ramakrishnan, Zhaocong Yuan, Jay Zhuo, Chen Feng et al.NeurIPS 2025 · 6 citations
- TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked TextSongshuo Lu, Hua Wang, Yutian Rong, Zhi Chen et al.EMNLP 2025 · 2 citations
- Talon: Breaking the Synchronization Barrier in Speculative Decoding with Hybrid Model-based and Retrieve-based DraftingXiangxiang Gao, Weisheng Xie, Lixin, Xuwei Fang et al.AAAI 2026
Builds on8
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
- EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
Related papers
- SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long SequencesJungyoub Cha, Hyunjong Kim, Sungzoon ChoACL Findings 2026
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative DecodingGuanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li et al.ICML 2025
- RACER: Retrieval-Augmented Contextual Rapid Speculative DecodingZihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang et al.ACL Findings 2026 · 1 citation
- DReSD: Dense Retrieval for Speculative DecodingMilan Gritta, Huiyin Xue, Gerasimos LampourasACL Findings 2025 · 1 citation
- Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMsHongyi Liu, Jiaji Huang, Zhen Jia, Youngsuk Park et al.ICLR 2026 · 5 citations
