Lune

ICLR2025Top-tier venue

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

Zihao Wang, Bin Cui, Shaoduo Gan

Abstract

Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KVcache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. However, most of these methods treat all layers equally, allocating the same KV budget to each layer. This approach is suboptimal, as some layers may be less sensitive to input tokens yet still receive the same budget as others. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions, i.e., sequence-wise and layer-wise. Based on our observations regarding layer-wise importance in inference, we propose SQUEEZEATTENTION to precisely optimize the allocation of KV-cache budget among layers on-the-fly and then incorporate three representative sequence-wise algorithms to compress the KV-cache for each layer with its very own budget. Specifically, we first measure each layer's importance by calculating the cosine similarity of the input prompt differences before and after the self-attention layers. Based on this similarity, we then categorize the layers into two groups and adjust their KV budgets accordingly. By optimizing the KV-cache from both sequence's and layer's dimensions, SQUEEZEATTENTION achieves around 30% to 70% of the memory reductions and up to 2.2 × of throughput improvements in a wide range of LLMs and benchmarks. The code is available at https://github.com/hetailang/SqueezeAttention.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Streaming Language Models with Attention Sinks cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Efficient Streaming Language Models with Attention Sinks
  • MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference cites Efficient Streaming Language Models with Attention Sinks
  • MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • EntroKV: Entropy-Guided Dynamic Budget Allocation for KV-Cache Compression cites Efficient Streaming Language Models with Attention Sinks
  • EntroKV: Entropy-Guided Dynamic Budget Allocation for KV-Cache Compression cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • SGLang: Efficient Execution of Structured Language Model Programs cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • SGLang: Efficient Execution of Structured Language Model Programs cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • PQCache: Product Quantization-based KVCache for Long Context LLM Inference cites Efficient Streaming Language Models with Attention Sinks
  • PQCache: Product Quantization-based KVCache for Long Context LLM Inference cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • PQCache: Product Quantization-based KVCache for Long Context LLM Inference cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. PQCache: Product Quantization-based KVCache for Long Context LLM InferenceHailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu et al.SIGMOD 2025 · 13 citations
  2. DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM ServingYing Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen et al.ICLR 2026 · 10 citations
  3. Enhancing Instruction Following of LLMs via Activation Steering with Dynamic RejectionMinjae Kang, Jaehyung KimICLR 2026 · 6 citations
  4. MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM InferenceWenxuan Zeng, Ye Dong, Jinjin Zhou, Jin Tan et al.NeurIPS 2025 · 4 citations
  5. EntroKV: Entropy-Guided Dynamic Budget Allocation for KV-Cache CompressionWenhao Gao, Haoran Cao, Yueyan Li, YongGao Xiao et al.ICML 2026

Builds on10

  1. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  2. SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun et al.NeurIPS 2024 · 1,586 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
  5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li et al.ICML 2023 · 683 citations

Related papers

  1. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM InferenceDongjie Yang, Xiaodong Han, Yan Gao, Yao Hu et al.ACL Findings 2024 · 31 citations
  2. ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token IdentificationYefei He, Luoming Zhang, Weijia Wu, Jing Liu et al.NeurIPS 2024 · 100 citations
  3. Layer-Condensed KV Cache for Efficient Inference of Large Language ModelsHaoyi Wu, Kewei TuACL 2024
  4. RazorAttention: Efficient KV Cache Compression Through Retrieval HeadsHanlin Tang, Yang Lin, Jing Lin, Qingsen Han et al.ICLR 2025
  5. KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV CacheFei Li, Song Liu, Weiguo Wu, Shiqiang Nie et al.AAAI 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines