Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao
Abstract
In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
204 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Language Models are Few-Shot Learners
- SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- SnapKV: LLM Knows What You are Looking for Before Generation cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
- Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
- LIMA: Less Is More for Alignment cites The Curious Case of Neural Text Degeneration
- Neural Attention Search cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- Neural Attention Search cites Language Models are Few-Shot Learners
- Neural Attention Search cites MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
- Neural Attention Search cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
- SGLang: Efficient Execution of Structured Language Model Programs cites Language Models are Few-Shot Learners
- SGLang: Efficient Execution of Structured Language Model Programs cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites SnapKV: LLM Knows What You are Looking for Before Generation
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
- Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time cites Language Models are Few-Shot Learners
Cited by top-tier papers204
- SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun et al.NeurIPS 2024 · 1,586 citations
- SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse AttentionHuiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu et al.NeurIPS 2024 · 479 citations
- Neural Attention SearchDifan Deng, Marius LindauerNeurIPS 2025 · 431 citations
Builds on15
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
- Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test TimeZichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang et al.NeurIPS 2023 · 557 citations
Related papers
- DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV CompactionYanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui et al.SOSP 2025
- RefreshKV: Updating Small KV Cache During Long-form GenerationFangyuan Xu, Tanya Goyal, Eunsol ChoiACL 2025 · 6 citations
- RazorAttention: Efficient KV Cache Compression Through Retrieval HeadsHanlin Tang, Yang Lin, Jing Lin, Qingsen Han et al.ICLR 2025
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM InferenceYuan Feng, Junlin Lv, Yukun Cao, Xike Xie et al.NeurIPS 2025 · 256 citations
- HitKV: Activation Frequency Knows Which Tokens Are ImportantSanle Zhao, Yujuan Tan, Jing Yu, Zhuoxin Bai et al.AAAI 2026
