Lune

ACL Findings2022

Attention Mechanism with Energy-Friendly Operations

Yu Wan, Baosong Yang, Dayiheng Liu, Rong Xiao, Derek F. Wong, Haibo Zhang, Boxing Chen, Lidia S. Chao

摘要

Attention mechanism has become the dominant module in natural language processing models.It is computationally intensive and depends on massive power-hungry multiplications.In this paper, we rethink variants of attention mechanism from the energy consumption aspects.After reaching the conclusion that the energy costs of several energyfriendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions.Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure.Code is available at: https://github.com/NLP2CT/E-Att.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation 引用了 Synthesizer: Rethinking Self-Attention for Transformer Models
  • Synthesizer: Rethinking Self-Attention for Transformer Models 引用了 Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
  • ShiftAddNet: A Hardware-Inspired Deep Network 引用了 AdderNet: Do We Really Need Multiplications in Deep Learning?
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper6

  1. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang 等EMNLP Findings 2020 · 1,787 次引用
  2. Synthesizer: Rethinking Self-Attention for Transformer ModelsYi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan 等ICML 2021 · 399 次引用
  3. ShiftAddNet: A Hardware-Inspired Deep NetworkHaoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li 等NeurIPS 2020 · 99 次引用
  4. Fixed Encoder Self-Attention Patterns in Transformer-Based Machine TranslationAlessandro Raganato, Yves Scherrer, Jörg TiedemannEMNLP Findings 2020 · 88 次引用
  5. AdderNet: Do We Really Need Multiplications in Deep Learning?Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi 等CVPR 2020

相关 Paper

  1. Softermax: Hardware/Software Co-Design of an Efficient Softmax for TransformersJacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany 等DAC 2021 · 143 次引用
  2. Adder Attention for Vision TransformerHan Shu, Jiahao Wang, Hanting Chen, Lin Li 等NeurIPS 2021 · 23 次引用
  3. On the Design of Novel Attention Mechanism for Enhanced Efficiency of TransformersSumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro VelasquezDAC 2024
  4. Selective Attention Improves TransformerYaniv Leviathan, Matan Kalman, Yossi MatiasICLR 2025
  5. A3: Accelerating Attention Mechanisms in Neural Networks with ApproximationTae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh 等HPCA 2020 · 241 次引用
黄昏的海面,两侧是细线勾勒的悬崖