Lune

EMNLP Findings2024

LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference

Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, Li Yuan

Abstract

Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs' KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge. In this work, we introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefilling phase, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. LOOK-M demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by 80% in some cases, it not only achieves up to 1.5x faster decoding but also maintains or even enhances performance across a variety of long context multimodal tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

43 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Streaming Language Models with Attention Sinks cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Efficient Streaming Language Models with Attention Sinks cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Efficient Streaming Language Models with Attention Sinks
  • Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression cites Efficient Streaming Language Models with Attention Sinks
  • Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression cites SnapKV: LLM Knows What You are Looking for Before Generation
  • SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
  • SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  • MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites Efficient Streaming Language Models with Attention Sinks
  • MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites Visual Instruction Tuning
  • MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites SnapKV: LLM Knows What You are Looking for Before Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers43

  1. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse AttentionHuiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu et al.NeurIPS 2024 · 479 citations
  2. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMsHaokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui et al.NeurIPS 2024 · 206 citations
  3. SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement LearningZhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang et al.NeurIPS 2025 · 63 citations
  4. StreamingTOM: Streaming Token Compression for Efficient Video UnderstandingXueyi Chen, Keda Tao, Kele Shao, Huan WangCVPR 2026 · 46 citations
  5. Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache CompressionKunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng HwangNeurIPS 2025 · 23 citations

Builds on23

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  4. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
  5. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations

Related papers

  1. Reducing Peak Memory Usage for Modern Multimodal Large Language Model PipelinesJunwan Kim, Hyunkyung BaeACL Findings 2026
  2. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language ModelsJinsong Shu, Chenyang Wu, Zhongle Xie, Baokun Wang et al.ACL Findings 2026
  3. HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model InferenceBowen Zeng, Feiyang Ren, Jun Zhang, Xiaoling Gu et al.ACL 2026 · 5 citations
  4. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context InferenceKunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode Wang et al.ACL 2025
  5. Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware ApproachYaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu et al.CVPR 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines