A-ViT: Adaptive Tokens for Efficient Vision Transformer
Hongxu Yin, Arash Vahdat, José M. Álvarez, Arun Mallya, Jan Kautz, Pavlo Molchanov
Abstract
We introduce A - ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A - ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds. We refor-mulate Adaptive Computation Time (ACT [17]) for this task, extending halting to discard redundant spatial tokens. The appealing architectural properties of vision transformers enables our adaptive token reduction mechanism to speed up inference without modifying the network architecture or inference hardware. We demonstrate that A - ViT requires no extra parameters or sub-network for halting, as we base the learning of adaptive halting on the original network parameters. We further introduce distributional prior regularization that stabilizes training compared to prior ACT approaches. On the image classification task (ImageNet1K), we show that our proposed A - ViT yields high efficacy in filtering informative spatial features and cutting down on the overall compute. The proposed method improves the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3% accuracy drop, outperforming prior art by a large margin.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
120 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MogaNet: Multi-order Gated Aggregation Network cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- MogaNet: Multi-order Gated Aggregation Network cites Focal Modulation Networks
- MogaNet: Multi-order Gated Aggregation Network cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- MogaNet: Multi-order Gated Aggregation Network cites Language Models are Few-Shot Learners
- MogaNet: Multi-order Gated Aggregation Network cites Training data-efficient image transformers & distillation through attention
- ProPainter: Improving Propagation and Transformer for Video Inpainting cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- Focal Modulation Networks cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Focal Modulation Networks cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Focal Modulation Networks cites Training data-efficient image transformers & distillation through attention
- Focal Modulation Networks cites SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Training data-efficient image transformers & distillation through attention
- Scale-Aware Modulation Meet Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scale-Aware Modulation Meet Transformer cites Focal Modulation Networks
- Scale-Aware Modulation Meet Transformer cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Scale-Aware Modulation Meet Transformer cites Training data-efficient image transformers & distillation through attention
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Cited by top-tier papers120
- Focal Modulation NetworksJianwei Yang, Chunyuan Li, Xiyang Dai, Jianfeng GaoNeurIPS 2022 · 494 citations
- Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and ResolutionMostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek et al.NeurIPS 2023 · 303 citations
- ProPainter: Improving Propagation and Transformer for Video InpaintingShangchen Zhou, Chongyi Li, Kelvin C. K. Chan, Chen Change LoyICCV 2023 · 205 citations
- Scale-Aware Modulation Meet TransformerWeifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang et al.ICCV 2023 · 154 citations
- MogaNet: Multi-order Gated Aggregation NetworkSiyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan et al.ICLR 2024 · 151 citations
Builds on26
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- SegFormer: Simple and Efficient Design for Semantic Segmentation with TransformersEnze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar et al.NeurIPS 2021 · 9,661 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
Related papers
- EViT: Expediting Vision Transformers via Token ReorganizationsYouwei Liang, Chongjian Ge, Zhan Tong, Yibing Song et al.ICLR 2022 · 137 citations
- Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionYulin Wang, Rui Huang, Shiji Song, Zeyi Huang et al.NeurIPS 2021 · 283 citations
- SPOT: Sparsification with Attention Dynamics via Token Relevance in Vision TransformersOded Schlesinger, Amirhossein Farzam, J. Matias Di Martino, Guillermo SapiroCVPR Findings 2026
- Dyna-ViT: Parameter-Free Pre-Encoder Token Pruning for Efficient Vision TransformersSyeda Fiza Rubab, Arslan Abdul Ghaffar, Malik Junaid Jami Gul, Sheriff Murtala, Ingyu Lee, Gyu Sang ChoiCVPR Findings 2026
- CF-ViT: A General Coarse-to-Fine Method for Vision TransformerMengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen et al.AAAI 2023 · 105 citations
