Lune

ACL Findings2025

TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models

Jaewoo Lee, Keyang Xuan, Chanakya Ekbote, Sandeep Polisetty, Yi R. Fung, Paul Pu Liang

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased model scale. While post-training pruning reduces model size in unimodal models, its application to MLLMs often yields limited success. Our analysis discovers that conventional methods fail to account for the unique token attributes across layers and modalities inherent to MLLMs. Inspired by this observation, we propose TAMP, a simple yet effective pruning framework tailored for MLLMs, featuring two key components: (1) Diversity-Aware Sparsity, which adjusts sparsity ratio per layer based on diversities among multimodal output tokens, preserving more parameters in highdiversity layers; and (2) Adaptive Multimodal Input Activation, which identifies representative multimodal input tokens using attention scores to guide unstructured weight pruning. We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for visionlanguage tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities. Empirical experiments across various multimodal evaluation benchmarks demonstrate that each component of our approach substantially outperforms existing pruning techniques.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs cites MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ReGATE: Learning Faster and Better with Fewer Tokens in MLLMsChaoyu Li, Yogesh Kulkarni, Pooyan FazliACL 2026
  2. Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language ModelsSijie Li, Biao Qian, Jungong HanCVPR 2026

Builds on23

  1. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
  2. SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
  3. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMsPeter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo et al.NeurIPS 2024 · 1,004 citations
  4. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
  5. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersZhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu et al.NeurIPS 2022 · 816 citations

Related papers

  1. Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language ModelsWeihao Ye, Qiong Wu, Wenhao Lin, Yiyi ZhouAAAI 2025 · 99 citations
  2. PrismPrune: Decoupling Saliency and Diversity in Attention for Efficient Visual Token Pruning in VLMsZiniu Liu, Shuheng Zhou, Mingqing Liu, Hao Deng, Huijia ZhuCVPR Findings 2026
  3. VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMsYingqi Fan, Anhao Zhao, Jinlan Fu, Junlong Tong et al.EMNLP 2025 · 11 citations
  4. DCP: Dual-Cue Pruning for Efficient Large Vision-Language ModelsLei Jiang, Zixun Zhang, Yuting Zeng, Chunzhao Xie et al.EMNLP 2025 · 2 citations
  5. Instruction-Guided Cross-Modal Clustering for Training-Free Visual Token Pruning in Vision-Language ModelsYunqian Yu, Biao Chen, Yunya Zhang, Tonglan Xie et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines