CVPR Findings2026
PrismPrune: Decoupling Saliency and Diversity in Attention for Efficient Visual Token Pruning in VLMs
Ziniu Liu, Shuheng Zhou, Mingqing Liu, Hao Deng, Huijia Zhu
Abstract
Vision-Language Models (VLMs) have achieved unprecedented performance by adopting high-resolution inputs, but at the cost of a quadratic surge in visual tokens that creates a severe computational bottleneck. Existing token pruning methods attempt to mitigate this, but struggle to balance saliency and diversity efficiently. Saliency-focused approaches discard vital context and visual diversity, while methods that incorporate diversity risk sacrificing essential salient tokens and are sensitive to initialization. This paper introduces PrismPrune, a novel and efficient pruning framework that resolves this dilemma. Our core insight is that the signals for saliency and diversity are already intrinsically decoupled within the multi-head attention mechanism. PrismPrune operationalizes this insight by decomposing the attention signal into its constituent components. First, it preserves the most salient tokens identified by the mean attention score. Second, it captures diverse tokens by leveraging attention variance within an efficient "filterthen-sample" strategy, which first filters for a high-potential candidate pool before performing diversity sampling exclusively on this subset. This hybrid approach thereby maintains a rich representation of both salient and contextual information. Extensive experiments demonstrate state-ofthe-art results. On LLaVA-1.5, we prune 77.8% of tokens with zero performance loss. For the high-resolution LLaVA-NeXT, we reduce pre-filling time by 10x and FLOPS by 13x while retaining over 91% of the original performance, significantly outperforming existing methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds on26
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
Related papers
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-DiversityZhengyao Fang, Pengyuan Lyu, Chengquan Zhang, Guangming Lu et al.ICLR 2026 · 25 citations
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMsJinhong Deng, Wen Li, Joey Tianyi Zhou, Yang HeNeurIPS 2025 · 23 citations
- CrisPrune: Combining Contextual Relevance and Intrinsic Saliency for Efficient Visual Token Pruning in MLLMsZiniu Liu, Shuheng Zhou, Mingqing Liu, Hao Deng et al.ACL Findings 2026
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language ModelsWeihao Ye, Qiong Wu, Wenhao Lin, Yiyi ZhouAAAI 2025 · 99 citations
- FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language ModelsJintao Tong, Wenwei Jin, Pengda Qin, Anqi Li et al.NeurIPS 2025 · 31 citations
