ACL Findings2026
Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
Abstract
Multimodal large language models (MLLMs) achieve strong visual-textual reasoning by scaling to high-resolution images and long video sequences, but this scalability introduces substantial inference-time memory overhead due to the growth of the key-value (KV) cache. Existing KV-cache compression methods primarily operate after the full multimodal context has been processed, and therefore do not address the peak memory consumption incurred during the prefill stage. We observe that visual tokens in MLLMs exhibit strong structural regularities and representational redundancy that can be exploited earlier in the inference pipeline. Based on this observation, we propose a sequential, structure-aware KV-cache compression framework that operates during prefill and enforces a fixed memory budget throughout input processing. Unlike conventional postprefill compression, which first constructs the full KV cache and compresses it afterward, our method compresses incrementally during prefix encoding. Experimental results show that our approach substantially reduces peak memory usage with minimal degradation in generative performance, enabling more practical and memory-efficient multimodal inference for large-scale visual inputs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Efficient Streaming Language Models with Attention Sinks cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Efficient Streaming Language Models with Attention Sinks
- Efficient Streaming Language Models with Attention Sinks cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Language Models are Few-Shot Learners
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Language Models are Few-Shot Learners
- SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
- SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Builds on18
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
Related papers
- HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model InferenceBowen Zeng, Feiyang Ren, Jun Zhang, Xiaoling Gu et al.ACL 2026 · 5 citations
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference AccelerationDezhan Tu, Danylo Vashchilenko, Yuzhe Lu, Panpan XuICLR 2025
- MMSep: Efficient Multimodal Long-Generation Reasoning via Multimodal Separator CompressionMingjie Ma, Yichao Ma, Jiannan Cao, Changhong Li et al.KDD 2026
- Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware ApproachYaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu et al.CVPR 2026 · 5 citations
- LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context InferenceZhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang et al.EMNLP Findings 2024 · 21 citations
