Lune

AAAI2025Top-tier venue

ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition

Seungdong Yoa, Seungjun Lee, Hye-Seung Cho, Bumsoo Kim, Woohyung Lim

Abstract

Vision Transformers (ViTs) have achieved remarkable success in various computer vision tasks. However, ViTs have a huge computational cost due to their inherent reliance on multi-head self-attention (MHSA), prompting efforts to accelerate ViTs for practical applications. To this end, recent works aim to reduce the number of tokens, mainly focusing on how to effectively prune or merge them. Nevertheless, since ViT tokens are generated from non-overlapping grid patches, they usually do not convey sufficient semantics, making it incompatible with efficient ViTs. To address this, we propose ImagePiece, a novel re-tokenization strategy for Vision Transformers. Following the MaxMatch strategy of NLP tokenization, ImagePiece groups semantically insufficient yet locally coherent tokens until they convey meaning. This simple retokenization is highly compatible with previous token reduction methods, being able to drastically narrow down relevant tokens, enhancing the inference speed of DeiT-S by 54% (nearly 1.5x faster) while achieving a 0.39% improvement in ImageNet classification accuracy. For hyper-speed inference scenarios (with 251% acceleration), our approach surpasses other baselines by an accuracy over 8%.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
  • Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
  • Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on28

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  4. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without ConvolutionsWenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan et al.ICCV 2021 · 4,909 citations
  5. Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetLi Yuan, Yunpeng Chen, Tao Wang, Weihao Yu et al.ICCV 2021 · 2,462 citations

Related papers

  1. EViT: Expediting Vision Transformers via Token ReorganizationsYouwei Liang, Chongjian Ge, Zhan Tong, Yibing Song et al.ICLR 2022 · 137 citations
  2. Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionYulin Wang, Rui Huang, Shiji Song, Zeyi Huang et al.NeurIPS 2021 · 283 citations
  3. HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision TransformersPeiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie et al.HPCA 2023 · 117 citations
  4. Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision TransformerYifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng et al.AAAI 2022 · 288 citations
  5. Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision TransformersSiyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines