EMNLP Findings2024
ATQ: Activation Transformation forWeight-Activation Quantization of Large Language Models
Abstract
There are many emerging quantization methods to resolve the problem that the huge demand on computational and storage costs hinders the deployment of Large language models (LLMs). However, their accuracy performance still can not satisfy the entire academic and industry community. In this work, we propose ATQ, an INT8 weight-activation quantization of LLMs, that can achieve almost lossless accuracy. We employ a mathematically equivalent transformation and a triangle inequality to constrain weight-activation quantization error to the sum of a weight quantization error and an activation quantization error. For the weight part, transformed weights are quantized along the in-feature dimension and the quantization error is compensated by optimizing following in-features. For the activation part, transformed activations are in the normal range and can be quantized easily. We provide comparison experiments to demonstrate that our ATQ method can achieve almost lossless in accuracy on OPT and LLaMA families in W8A8 quantization settings. The increase of perplexity is within 1 and the accuracy degradation is within 0.5 percent even in the worst case.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- QLoRA: Efficient Finetuning of Quantized LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models cites Language Models are Few-Shot Learners
- Energy-Efficient and Dequantization-Free Quantization of LLMs: A Spiking Neural Network Approach to Salient Value Mitigation cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot cites Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
Cited by top-tier papers1
Builds on7
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
- Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningElias Frantar, Dan AlistarhNeurIPS 2022 · 440 citations
Related papers
- LQER: Low-Rank Quantization Error Reconstruction for LLMsCheng Zhang, Jianyi Cheng, George Anthony Constantinides, Yiren ZhaoICML 2024 · 33 citations
- "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM QuantizationEldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz et al.ACL 2025
- MobileQuant: Mobile-friendly Quantization for On-device Language ModelsFuwen Tan, Royson Lee, Lukasz Dudziak, Shell Xu Hu et al.EMNLP Findings 2024 · 13 citations
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank CompensationZhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn et al.AAAI 2024 · 50 citations
- ACBQ: Adaptive Cross-Block Quantization of Large Language ModelsHailing Wang, Jianglin Lu, Yitian Zhang, Huimin Zeng et al.ACL 2026
