EMNLP Findings2024
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv, Yi Liu
Abstract
Large Language Models (LLMs) have demonstrated exceptional proficiency in languagerelated tasks, but their deployment poses significant challenges due to substantial memory and storage requirements. Weight-only quantization has emerged as a promising solution, significantly reducing memory and storage needs without sacrificing too much performance. In this study, we introduceSign-Round, a method that leverages signed gradient descent (SignSGD) to optimize rounding values and weight clipping in just 200 steps. SignRound integrates the advantages of Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ), delivering exceptional results across 2 to 4 bits while minimizing tuning costs and avoiding additional inference overhead. For example, SignRound achieved absolute average accuracy improvements ranging from 6.91% to 33.22% at 2 bits, as measured by the average zero-shot accuracy across 11 tasks. It also demonstrates strong generalization in recent models, achieving near-lossless 4-bit quantization in most scenarios. The source code is publicly available at https://github.com/intel/auto-round.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Measuring Massive Multitask Language Understanding cites PIQA: Reasoning about Physical Commonsense in Natural Language
- FP4 All the Way: Fully Quantized Training of Large Language Models cites QLoRA: Efficient Finetuning of Quantized LLMs
- QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs cites Measuring Massive Multitask Language Understanding
- Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization cites PIQA: Reasoning about Physical Commonsense in Natural Language
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
- Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites PIQA: Reasoning about Physical Commonsense in Natural Language
Cited by top-tier papers11
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language ModelsJing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong et al.ICLR 2024 · 75 citations
- AffineQuant: Affine Transformation Quantization for Large Language ModelsYuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling et al.ICLR 2024 · 56 citations
- Qronos: Correcting the Past by Shaping the Future... in Post-Training QuantizationShihao Zhang, Haoyu Zhang, Ian Colbert, Rayan SaabICLR 2026 · 27 citations
- Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache QuantizationMinsu Kim, Seongmin Hong, Ryeowook Ko, Soongyu Choi et al.ISCA 2025 · 17 citations
- FP4 All the Way: Fully Quantized Training of Large Language ModelsBrian Chmiel, Maxim Fishman, Ron Banner, Daniel SoudryNeurIPS 2025 · 9 citations
Builds on26
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
Related papers
- GuidedQuant: Large Language Model Quantization via Exploiting End Loss GuidanceJinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens J. S. Schaefer et al.ICML 2025
- FlexRound: Learnable Rounding based on Element-wise Division for Post-Training QuantizationJung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, Dongsoo LeeICML 2023 · 56 citations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
- SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language ModelsHan Liu, Haotian Gao, Xiaotong Zhang, Changya Li et al.KDD 2025 · 1 citation
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language ModelsWenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu et al.ICLR 2024 · 395 citations
