Lune

ACL Findings2026

FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration

Dongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon Kim

Abstract

While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and keyvalue (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82× in prefill and 2.87× in decoding compared to the full-context baseline, while matching the accuracy of the decodingonly baselines. Our code is available at https: //github.com/dongwonjo/FastKV.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Efficient Streaming Language Models with Attention Sinks
  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
  • MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering cites Efficient Streaming Language Models with Attention Sinks
  • SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
  • FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • DesireKV: Decoupling Sensitivity and Importance for Reasoning-Aware KV Cache Compression cites KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  • DesireKV: Decoupling Sensitivity and Importance for Reasoning-Aware KV Cache Compression cites SnapKV: LLM Knows What You are Looking for Before Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-AnsweringJunbin Xiao, Jiajun Chen, Tianxiang Sun, Xun Yang et al.CVPR 2026 · 6 citations
  2. DesireKV: Decoupling Sensitivity and Importance for Reasoning-Aware KV Cache CompressionPengyu Cheng, Jiacheng Wang, Tianle Chen, Bei Liu et al.AAAI 2026

Builds on13

  1. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  2. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
  5. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionJay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar et al.NeurIPS 2024 · 727 citations

Related papers

  1. Adaptive Layer Selection for Layer-Wise Token Pruning in LLM InferenceRei Taniguchi, Yuyang Dong, Makoto Onizuka, Chuan XiaoACL Findings 2026
  2. StructKV: Preserving the Structural Skeleton for Scalable Long-Context InferenceZhirui Chen, Peiyang Liu, Ling ShaoACL Findings 2026
  3. TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache SelectionWei Wu, Zhuoshi Pan, Kun Fu, Chao Wang et al.EMNLP 2025 · 2 citations
  4. SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal BudgetZihao Wang, Bin Cui, Shaoduo GanICLR 2025
  5. GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache EvictionXuelin Li, Xiangqi Jin, Linfeng ZhangEMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines