ACL Findings2021
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter A. Milder, H. Andrew Schwartz, Niranjan Balasubramanian
Abstract
How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accuracies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This informs the design of an inference-time quantization technique using both pruning and logscaled mapping which produces only a few (e.g. 2 3 ) unique values. Over the tasks of question answering and sentiment analysis, we find nearly 80% of attention values can be pruned to zeros with minimal (< 1.0%) relative loss in accuracy. We use this pruning technique in conjunction with quantizing the attention values to only a 3-bit format, without retraining, resulting in only a 0.8% accuracy reduction on question answering with fine-tuned RoBERTa.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers3
- Towards Accurate Post-Training Quantization for Vision TransformerYifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai et al.ACM MM 2022 · 68 citations
- Hallucination Detection in LLMs with Topological Divergence on Attention GraphsAlexandra Bazarova, Andrei Volodichev, Aleksandr Yugay, Andrey Shulga et al.ACL 2026 · 14 citations
- D2 Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution AwarenessLang Xiong, Ning Liu, Ao Ren, Yuheng Bai et al.AAAI 2026
Builds on12
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTSheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma et al.AAAI 2020 · 656 citations
- Hopfield Networks is All You NeedHubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl et al.ICLR 2021 · 620 citations
- I-BERT: Integer-only BERT QuantizationSehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney et al.ICML 2021 · 439 citations
- The Lottery Ticket Hypothesis for Pre-trained BERT NetworksTianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu et al.NeurIPS 2020 · 428 citations
- ETC: Encoding Long and Structured Inputs in TransformersJoshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek et al.EMNLP 2020 · 268 citations
Related papers
- A Fast Post-Training Pruning Framework for TransformersWoosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun et al.NeurIPS 2022 · 247 citations
- General Purpose Text Embeddings from Pre-trained Language Models for Scalable InferenceJingfei Du, Myle Ott, Haoran Li, Xing Zhou et al.EMNLP Findings 2020 · 8 citations
- Masking as an Efficient Alternative to Finetuning for Pretrained Language ModelsMengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi et al.EMNLP 2020 · 61 citations
- 8-bit Transformer Inference and Fine-tuning for Edge AcceleratorsJeffrey Yu, Kartik Prabhu, Yonatan Urman, Robert M. Radway et al.ASPLOS 2024 · 26 citations
- PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language ModelsZhuocheng Gong, Jiahao Liu, Qifan Wang, Yang Yang et al.ACL Findings 2023 · 5 citations
