Lune

ICCV2023Top-tier venue

Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation

Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, Yifan Liu

Abstract

Vision transformers have achieved leading performance on various visual tasks yet still suffer from high computational complexity. The situation deteriorates in dense prediction tasks like semantic segmentation, as high-resolution inputs and outputs usually imply more tokens involved in computations. Directly removing the less attentive tokens has been discussed for the image classification task but can not be extended to semantic segmentation since a dense prediction is required for every patch. To this end, this work introduces a Dynamic Token Pruning (DToP) method based on the early exit of tokens for semantic segmentation. Motivated by the coarse-to-fine segmentation process by humans, we naturally split the widely adopted auxiliary-loss-based network architecture into several stages, where each auxiliary block grades every token’s difficulty level. We can finalize the prediction of easy tokens in advance without completing the entire forward pass. Moreover, we keep k highest confidence tokens for each semantic category to uphold the representative context information. Thus, computational complexity will change with the difficulty of the input, akin to the way humans do segmentation. Experiments suggest that the proposed DToP architecture reduces on average 20% ∼ 35% of computational cost for current semantic segmentation methods based on plain vision transformers without accuracy degradation. The code is available through the following link: https://github.com/zbwxp/Dynamic-Token-Pruning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

21 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable cites Zero-TPrune: Zero-Shot Token Pruning Through Leveraging of the Attention Graph in Pre-Trained Transformers
  • Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
  • A ConvNet for the 2020s cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
  • ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers cites SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
  • ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers cites Zero-TPrune: Zero-Shot Token Pruning Through Leveraging of the Attention Graph in Pre-Trained Transformers
  • Zero-TPrune: Zero-Shot Token Pruning Through Leveraging of the Attention Graph in Pre-Trained Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers21

  1. SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch NormalizationJialong Guo, Xinghao Chen, Yehui Tang, Yunhe WangICML 2024 · 40 citations
  2. Zero-TPrune: Zero-Shot Token Pruning Through Leveraging of the Attention Graph in Pre-Trained TransformersHongjie Wang, Bhishma Dedhia, Niraj K. JhaCVPR 2024 · 24 citations
  3. ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision TransformersNarges Norouzi, Svetlana Orlova, Daan de Geus, Gijs DubbelmanCVPR 2024 · 15 citations
  4. Co-Me: Confidence Guided Token Merging for Visual Geometric TransformersYutian Chen, Yuheng Qiu, Ruogu Li, Jay Patrikar et al.CVPR 2026 · 9 citations
  5. Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and PrunableLizhen Xu, Zehao Wu, Wenzhao Qiu, Shanmin Pang et al.AAAI 2026 · 6 citations

Builds on16

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. SegFormer: Simple and Efficient Design for Semantic Segmentation with TransformersEnze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar et al.NeurIPS 2021 · 9,661 citations
  3. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  4. A ConvNet for the 2020sZhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer et al.CVPR 2022 · 6,782 citations
  5. Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetLi Yuan, Yunpeng Chen, Tao Wang, Weihao Yu et al.ICCV 2021 · 2,462 citations

Related papers

  1. DynamicViT: Efficient Vision Transformers with Dynamic Token SparsificationYongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu et al.NeurIPS 2021 · 1,343 citations
  2. Token Cropr: Faster ViTs for Quite a Few TasksBenjamin Bergner, Christoph Lippert, Aravindh MahendranCVPR 2025
  3. Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision TransformersSifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang et al.CVPR 2023
  4. A Simple Romance Between Multi-Exit Vision Transformer and Token ReductionDongyang Liu, Meina Kan, Shiguang Shan, Xilin ChenICLR 2024 · 12 citations
  5. Frequency-Aware Token Reduction for Efficient Vision TransformerDongJae Lee, Jiwan Hur, Jaehyun Choi, Jaemyung Yu et al.NeurIPS 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines