Lune

ACL Findings2025

Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?

Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, Linfeng Zhang

Abstract

Multimodal large language models (MLLMs) have shown remarkable performance for crossmodal understanding and generation, yet still suffer from severe inference costs. Recently, abundant works have been proposed to solve this problem with token pruning, which identifies the redundant tokens in MLLMs and then prunes them to reduce the computation and KV storage costs, leading to significant acceleration without training. While these methods claim efficiency gains, critical questions about their fundamental design and evaluation remain unanswered: Why do many existing approaches underperform even compared to naive random token selection? Are attention-based scoring sufficient for reliably identifying redundant tokens? Is language information really helpful during token pruning? What makes a good trade-off between token importance and duplication? Are current evaluation protocols comprehensive and unbiased? The ignorance of previous research on these problems hinders the long-term development of token pruning. In this paper, we answer these questions one by one, providing insights into the design of future token pruning methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

24 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs cites Visual Instruction Tuning
  • Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Language Models are Few-Shot Learners
  • Variation-aware Vision Token Dropping for Faster Large Vision-Language Models cites Visual Instruction Tuning
  • Variation-aware Vision Token Dropping for Faster Large Vision-Language Models cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Variation-aware Vision Token Dropping for Faster Large Vision-Language Models cites Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
  • Efficient Multi-modal Large Language Models via Progressive Consistency Distillation cites Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
  • Visual Instruction Tuning cites Language Models are Few-Shot Learners
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation cites Visual Instruction Tuning
  • Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation cites Learning Transferable Visual Models From Natural Language Supervision
  • Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation cites Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
  • Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering cites Language Models are Few-Shot Learners
  • Efficient Multi-modal Large Language Models via Progressive Consistency Distillation cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Efficient Multi-modal Large Language Models via Progressive Consistency Distillation cites Visual Instruction Tuning
  • Efficient Multi-modal Large Language Models via Progressive Consistency Distillation cites Learning Transferable Visual Models From Natural Language Supervision
  • Efficient Multi-modal Large Language Models via Progressive Consistency Distillation cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention cites Visual Instruction Tuning
  • Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers24

  1. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMsQizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu et al.NeurIPS 2025 · 104 citations
  2. Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact ExplanationSiwei Wen, Junyan Ye, Peilin Feng, Hengrui Kang et al.NeurIPS 2025 · 82 citations
  3. Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context RetentionXin Zou, Di Lu, Yizhou Wang, Yibo Yan et al.NeurIPS 2025 · 49 citations
  4. Efficient Multi-modal Large Language Models via Progressive Consistency DistillationZichen Wen, Shaobo Wang, Yufa Zhou, Junyuan Zhang et al.NeurIPS 2025 · 26 citations
  5. Variation-aware Vision Token Dropping for Faster Large Vision-Language ModelsChen junjie, Xuyang Liu, Zichen Wen, Yiyu Wang et al.CVPR 2026 · 24 citations

Builds on17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  5. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations

Related papers

  1. Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters MoreZichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang et al.EMNLP 2025 · 4 citations
  2. TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal ModelsHao Zhang, Mengsi Lyu, Chenrui He, Yulong Ao et al.ACL Findings 2026 · 1 citation
  3. Skip-It? Theoretical Conditions for Layer Skipping in Vision–Language ModelsMax Hartman, Vidhata Jayaraman, Moulik Choraria, Akhil Bhimaraju et al.ICML 2026 · 1 citation
  4. Reducing Token Redundancy in LVLMs: A Systematic Review of Token Pruning MethodsHanzhang Yuan, Mengxuan Hu, Wenhao Zhang, Tianlong Wang et al.ACL 2026
  5. HAWK: Head Importance-Aware Visual Token Pruning in Multimodal ModelsQihui Zhu, Tao Zhang, Yuchen Wang, Shuangwu Chen et al.CVPR 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines