ACL Findings2022
Attention Mechanism with Energy-Friendly Operations
Yu Wan, Baosong Yang, Dayiheng Liu, Rong Xiao, Derek F. Wong, Haibo Zhang, Boxing Chen, Lidia S. Chao
Abstract
Attention mechanism has become the dominant module in natural language processing models.It is computationally intensive and depends on massive power-hungry multiplications.In this paper, we rethink variants of attention mechanism from the energy consumption aspects.After reaching the conclusion that the energy costs of several energyfriendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions.Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure.Code is available at: https://github.com/NLP2CT/E-Att.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation cites Synthesizer: Rethinking Self-Attention for Transformer Models
- Synthesizer: Rethinking Self-Attention for Transformer Models cites Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
- ShiftAddNet: A Hardware-Inspired Deep Network cites AdderNet: Do We Really Need Multiplications in Deep Learning?
Builds on6
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Synthesizer: Rethinking Self-Attention for Transformer ModelsYi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan et al.ICML 2021 · 399 citations
- ShiftAddNet: A Hardware-Inspired Deep NetworkHaoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li et al.NeurIPS 2020 · 99 citations
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine TranslationAlessandro Raganato, Yves Scherrer, Jörg TiedemannEMNLP Findings 2020 · 88 citations
- AdderNet: Do We Really Need Multiplications in Deep Learning?Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi et al.CVPR 2020
Related papers
- Softermax: Hardware/Software Co-Design of an Efficient Softmax for TransformersJacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany et al.DAC 2021 · 143 citations
- Adder Attention for Vision TransformerHan Shu, Jiahao Wang, Hanting Chen, Lin Li et al.NeurIPS 2021 · 23 citations
- On the Design of Novel Attention Mechanism for Enhanced Efficiency of TransformersSumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro VelasquezDAC 2024
- Selective Attention Improves TransformerYaniv Leviathan, Matan Kalman, Yossi MatiasICLR 2025
- A3: Accelerating Attention Mechanisms in Neural Networks with ApproximationTae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh et al.HPCA 2020 · 241 citations
