Lune

ICLR2025Top-tier venue

Training Free Exponential Context Extension via Cascading KV Cache

Jeffrey Willette, Heejun Lee, Youngwan Lee, Myeongjae Jeon, Sung Ju Hwang

Abstract

The transformer's context window is vital for tasks such as few-shot learning and conditional generation as it preserves previous tokens for active memory. However, as the context lengths increase, the computational costs grow quadratically, hindering the deployment of large language models (LLMs) in real-world, long sequence scenarios. Although some recent key-value caching (KV Cache) methods offer linear inference complexity, they naively manage the stored context, prematurely evicting tokens and losing valuable information. Moreover, they lack an optimized prefill/prompt stage strategy, resulting in higher latency than even quadratic attention for realistic context sizes. In response, we introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens, enabling the model to maintain longer context histories without increasing the cache size. Our approach outperforms linear caching baselines across key benchmarks, including streaming perplexity, question answering, book summarization, and passkey retrieval, where it retains better retrieval accuracy at 1M tokens after four doublings of the cache size of 65K. Additionally, our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens. These innovations not only enhance the computational efficiency of LLMs but also pave the way for their effective deployment in resource-constrained environments, enabling large-scale, real-time applications with significantly reduced latency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Streaming Language Models with Attention Sinks cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Efficient Streaming Language Models with Attention Sinks
  • Efficient Streaming Language Models with Attention Sinks cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • Efficient Streaming Language Models with Attention Sinks cites Reformer: The Efficient Transformer
  • V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding cites Efficient Streaming Language Models with Attention Sinks
  • V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding cites Reformer: The Efficient Transformer
  • Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings cites Efficient Streaming Language Models with Attention Sinks
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Reformer: The Efficient Transformer
  • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites Reformer: The Efficient Transformer
  • SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
  • SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • SnapKV: LLM Knows What You are Looking for Before Generation cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • AttentionPredictor: Temporal Patterns Matter for KV Cache Compression cites Efficient Streaming Language Models with Attention Sinks
  • AttentionPredictor: Temporal Patterns Matter for KV Cache Compression cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • AttentionPredictor: Temporal Patterns Matter for KV Cache Compression cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • AttentionPredictor: Temporal Patterns Matter for KV Cache Compression cites SnapKV: LLM Knows What You are Looking for Before Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Autoregressive Adversarial Post-Training for Real-Time Interactive Video GenerationShanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang et al.NeurIPS 2025 · 89 citations
  2. Attention Sinks: A 'Catch, Tag, Release' Mechanism for EmbeddingsStephen Zhang, Mustafa Khan, Vardan PapyanNeurIPS 2025 · 18 citations
  3. AttentionPredictor: Temporal Patterns Matter for KV Cache CompressionQingyue Yang, Jie Wang, Xing Li, Zhihai Wang et al.NeurIPS 2025 · 8 citations
  4. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position EncodingJunqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu et al.ICCV 2025 · 5 citations
  5. ACON: Optimizing Context Compression for Long-horizon LLM AgentsMinki Kang, Wei-Ning Chen, Dongge Han, Huseyin Inan et al.ICML 2026

Builds on13

  1. Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
  2. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
  3. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  4. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
  5. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations

Related papers

  1. RefreshKV: Updating Small KV Cache During Long-form GenerationFangyuan Xu, Tanya Goyal, Eunsol ChoiACL 2025 · 6 citations
  2. LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional EncodingHaocheng Xia, Mihir Pamnani, Hanxi Fang, Supawit Chockchowwat et al.ICML 2026
  3. Layer-Condensed KV Cache for Efficient Inference of Large Language ModelsHaoyi Wu, Kewei TuACL 2024
  4. KVzip: Query-Agnostic KV Cache Compression with Context ReconstructionJang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee et al.NeurIPS 2025 · 103 citations
  5. LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without GenerationJinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim et al.ICLR 2026 · 8 citations
Dusk over the sea between two cliffs drawn in fine vertical lines