ACL Findings2022
Attention Mechanism with Energy-Friendly Operations
Yu Wan, Baosong Yang, Dayiheng Liu, Rong Xiao, Derek F. Wong, Haibo Zhang, Boxing Chen, Lidia S. Chao
摘要
Attention mechanism has become the dominant module in natural language processing models.It is computationally intensive and depends on massive power-hungry multiplications.In this paper, we rethink variants of attention mechanism from the energy consumption aspects.After reaching the conclusion that the energy costs of several energyfriendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions.Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure.Code is available at: https://github.com/NLP2CT/E-Att.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation 引用了 Synthesizer: Rethinking Self-Attention for Transformer Models
- Synthesizer: Rethinking Self-Attention for Transformer Models 引用了 Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
- ShiftAddNet: A Hardware-Inspired Deep Network 引用了 AdderNet: Do We Really Need Multiplications in Deep Learning?
它引用的 Paper6
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang 等EMNLP Findings 2020 · 1,787 次引用
- Synthesizer: Rethinking Self-Attention for Transformer ModelsYi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan 等ICML 2021 · 399 次引用
- ShiftAddNet: A Hardware-Inspired Deep NetworkHaoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li 等NeurIPS 2020 · 99 次引用
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine TranslationAlessandro Raganato, Yves Scherrer, Jörg TiedemannEMNLP Findings 2020 · 88 次引用
- AdderNet: Do We Really Need Multiplications in Deep Learning?Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi 等CVPR 2020
相关 Paper
- Softermax: Hardware/Software Co-Design of an Efficient Softmax for TransformersJacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany 等DAC 2021 · 143 次引用
- Adder Attention for Vision TransformerHan Shu, Jiahao Wang, Hanting Chen, Lin Li 等NeurIPS 2021 · 23 次引用
- On the Design of Novel Attention Mechanism for Enhanced Efficiency of TransformersSumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro VelasquezDAC 2024
- Selective Attention Improves TransformerYaniv Leviathan, Matan Kalman, Yossi MatiasICLR 2025
- A3: Accelerating Attention Mechanisms in Neural Networks with ApproximationTae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh 等HPCA 2020 · 241 次引用
