Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, Anand Raghunathan
摘要
Transformers have transformed the field of natural language processing. Their superior performance is largely attributed to the use of stacked “self-attention” layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了最后一个公式，引用它的顶会 Paper 也一样。你提问，回答直接引用原文。
引用它的顶会 Paper21
- I-ViT: Integer-only Quantization for Efficient Vision Transformer InferenceZhikai Li, Qingyi GuICCV 2023 · 被引用 176 次
- NN-LUT: neural approximation of non-linear operations for efficient transformer inferenceJoonsang Yu, Junki Park, Seongmin Park, Minsoo Kim 等DAC 2022 · 被引用 63 次
- Accelerating attention through gradient-based learned runtime pruningZheng Li, Soroush Ghodrati, Amir Yazdanbakhsh, Hadi Esmaeilzadeh 等ISCA 2022 · 被引用 48 次
- Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip RecomputationAmir Yazdanbakhsh, Ashkan Moradifirouzabadi, Zheng Li, Mingu KangMICRO 2022 · 被引用 47 次
- SALO: an efficient spatial accelerator enabling hybrid sparse attention mechanisms for long sequencesGuan Shen, Jieru Zhao, Quan Chen, Jingwen Leng 等DAC 2022 · 被引用 37 次
它引用的顶会 Paper2
相关 Paper
- MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear FunctionsNeo Wei Ming, Zhehui Wang, Cheng Liu, Rick Siow Mong Goh 等ICLR 2023
- On the Design of Novel Attention Mechanism for Enhanced Efficiency of TransformersSumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro VelasquezDAC 2024
- ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural NetworksTae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim 等ISCA 2021 · 被引用 185 次
- A length adaptive algorithm-hardware co-design of transformer on FPGA through sparse attention and dynamic pipeliningHongwu Peng, Shaoyi Huang, Shiyang Chen, Bingbing Li 等DAC 2022 · 被引用 49 次
- FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation PredictionYubin Qin, Yang Wang, Dazheng Deng, Zhiren Zhao 等ISCA 2023 · 被引用 113 次
