Lune

ICLR2025Top-tier venue

Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection

Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yinxiao Liu, Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng, Bang Liu, Jindong Chen

Abstract

Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing a dramatic increase in latency. In this paper, we propose a novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Specifically, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. It is notable that Sparse RAG combines the assessment of each individual document and the generation of the response into a single process. The designed sparse mechanism in a RAG system can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, filtering out undesirable contexts enhances the model's focus on relevant context, inherently improving its generation quality. Evaluation results on four datasets show that Sparse RAG can be used to strike an optimal balance between generation quality and computational efficiency, demonstrating its generalizability across tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

10 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • TS-Memory: Plug-and-Play Memory for Time Series Foundation Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • TS-Memory: Plug-and-Play Memory for Time Series Foundation Models cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Influence Guided Context Selection for Effective Retrieval-Augmented Generation cites LoRA: Low-Rank Adaptation of Large Language Models
  • Influence Guided Context Selection for Effective Retrieval-Augmented Generation cites Provence: efficient and robust context pruning for retrieval-augmented generation
  • Influence Guided Context Selection for Effective Retrieval-Augmented Generation cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Provence: efficient and robust context pruning for retrieval-augmented generation cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers10

  1. Influence Guided Context Selection for Effective Retrieval-Augmented GenerationJiale Deng, Yanyan Shen, Ziyuan Pei, Youmin Chen et al.NeurIPS 2025 · 8 citations
  2. Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language ModelsHaoyu Wang, Peihao Wang, Mufei Li, Shikun Liu et al.NeurIPS 2025 · 6 citations
  3. TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked TextSongshuo Lu, Hua Wang, Yutian Rong, Zhi Chen et al.EMNLP 2025 · 2 citations
  4. Provence: efficient and robust context pruning for retrieval-augmented generationNadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane ClinchantICLR 2025 · 2 citations
  5. TS-Memory: Plug-and-Play Memory for Time Series Foundation ModelsSisuo Lyu, Siru Zhong, Tiegang Chen, Weilin Ruan et al.KDD 2026

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  5. Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations

Related papers

  1. Incentivizing Retrieval-Augmented Generation via Inner Adaptive Context SelectionChenxu Cui, Lin Shen, Haihui Fan, Sa Zhu et al.SIGIR 2026
  2. RAPID: Long-Context Inference with Retrieval-Augmented Speculative DecodingGuanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li et al.ICML 2025
  3. METIS: Fast Quality-Aware RAG Systems with Configuration AdaptationSiddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du et al.SOSP 2025 · 3 citations
  4. The Power of Noise: Redefining Retrieval for RAG SystemsFlorin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice et al.SIGIR 2024 · 212 citations
  5. ECoRAG: Evidentiality-guided Compression for Long Context RAGYeonseok Jeong, Jinsu Kim, Dohyeon Lee, Seung-won HwangACL Findings 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines