Lune

ACL Findings2025

MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference

Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, Minjia Zhang

Abstract

State-of-the-art 2-bit KV cache quantization techniques achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks. However, further pushing the compression ratio fails to deliver performance gains. In this work, we revisit these approaches by considering, additionally, adaptive KV methods that retain LLM accuracy with only a subset of KV states. This leads us to propose a method based on 2-bit KV cache quantization with adaptive KV policies. In addition, we take an algorithm and system co-design approach by developing hardwarefriendly kernels to accelerate LLM inference while making MiniKV compatible with existing memory-efficient attention techniques such as FlashAttention, effectively translating algorithmic improvements into system performance gains. Experiments on a wide range of long context tasks show that MiniKV effectively achieves >80% KV cache compression while retaining accuracy, outperforming state-of-theart methods while achieving excellent latency, throughput, and memory consumption improvements in long context inference.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
  • C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference cites KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  • C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference cites SnapKV: LLM Knows What You are Looking for Before Generation
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • SnapKV: LLM Knows What You are Looking for Before Generation cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM InferenceChuheng Du, Junyi Chen, Hanlin Tang, Kan Liu et al.KDD 2026 · 3 citations

Builds on13

  1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  2. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
  3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
  4. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
  5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li et al.ICML 2023 · 683 citations

Related papers

  1. JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context InferenceChengyu Sun, Yaqi Xia, Hulin Wang, Donglin Yang et al.PPoPP 2026 · 1 citation
  2. ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token IdentificationYefei He, Luoming Zhang, Weijia Wu, Jing Liu et al.NeurIPS 2024 · 100 citations
  3. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language ModelsAlina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev et al.ICML 2025
  4. Self-Indexing KVCache: Predicting Sparse Attention from Compressed KeysXu Yang, Jiapeng Zhang, Dongyang Zhao, Guo Chen et al.AAAI 2026
  5. STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank ControlPriyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang, SeungJae Lee et al.ICML 2026 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines