PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer
Pierre-David Letourneau, Manish Kumar Singh, Hsin-Pai Cheng, Shizhong Han, Yunxiao Shi, Dalton Jones, Matthew Harper Langston, Hong Cai, Fatih Porikli
摘要
We present Polynomial Attention Drop-in Replacement (PADRe), a novel and unifying framework designed to replace the conventional self-attention mechanism in transformer models. Notably, several recent alternative attention mechanisms, including Hyena, Mamba, SimA, Conv2Former, and Castling-ViT, can be viewed as specific instances of our PADRe framework. PADRe leverages polynomial functions and draws upon established results from approximation theory, enhancing computational efficiency without compromising accuracy. PADRe's key components include multiplicative nonlinearities, which we implement using straightforward, hardware-friendly operations such as Hadamard products, incurring only linear computational and memory costs. PADRe further avoids the need for using complex functions such as Softmax, yet it maintains comparable or superior accuracy compared to traditional self-attention. We assess the effectiveness of PADRe as a drop-in replacement for self-attention across diverse computer vision tasks. These tasks include image classification, image-based 2D object detection, and 3D point cloud object detection. Empirical results demonstrate that PADRe runs significantly faster than the conventional self-attention (11× ∼ 43× faster on server GPU and mobile NPU) while maintaining similar accuracy when substituting self-attention in the transformer models.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 2 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Segment Anything 引用了 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers 引用了 Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
- Visual Instruction Tuning 引用了 High-Resolution Image Synthesis with Latent Diffusion Models
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows 引用了 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
引用它的顶会 Paper2
它引用的 Paper22
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn 等ICLR 2021 · 21,477 次引用
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao 等ICCV 2023 · 13,211 次引用
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser 等CVPR 2022 · 13,123 次引用
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
相关 Paper
- Hyena Hierarchy: Towards Larger Convolutional Language ModelsMichael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu 等ICML 2023 · 481 次引用
- Fundamental Limitations on Subquadratic Alternatives to TransformersJosh Alman, Hantao YuICLR 2025
- A 2-Dimensional State Space Layer for Spatial Inductive BiasEthan Baron, Itamar Zimerman, Lior WolfICLR 2024 · 18 次引用
- KDEformer: Accelerating Transformers via Kernel Density EstimationAmir Zandieh, Insu Han, Majid Daliri, Amin KarbasiICML 2023 · 55 次引用
- Adder Attention for Vision TransformerHan Shu, Jiahao Wang, Hanting Chen, Lin Li 等NeurIPS 2021 · 23 次引用
