ACL Findings2026
Break Through the Compression Bottleneck: From Theory to Practice
Xiusheng Huang, Lu Wang, Yequan Wang, Jun Zhao, Kang Liu
Abstract
As the parameter size of language models continues to grow, effective model compression is required to reduce their computational and memory overhead. Existing compression methods suffer from bottleneck issues: when the compression ratio is increased, performance degrades significantly. Low-rank decomposition and quantization are two prominent compression methods that have been proven to significantly reduce the computational and memory requirements of Large Language Models (LLMs) while maintaining model accuracy. Evidently, combining these two methods will break through the existing compression bottleneck. However, how these two methods interact when combined remains a critical question for developers, as many assume they are orthogonal, meaning their combination would not introduce additional errors beyond those independently introduced by each method. This paper provides the first mathematical proof that lowrank decomposition and quantization are nonorthogonal. We validate these findings through a series of experiments on large language models. Our results demonstrate that these methods are non-orthogonal, and their combination leads to significant performance degradation. Importantly, we propose a novel approach Diagonal Adhesive Method (DAM), which can effectively combine the two methods and mitigate the performance loss. Our research provides deep insights into model compression and lays a solid theoretical and experimental foundation for future related studies.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs cites PIQA: Reasoning about Physical Commonsense in Natural Language
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs cites QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Builds on14
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsSaleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li et al.NeurIPS 2024 · 723 citations
- QuIP: 2-Bit Quantization of Large Language Models With GuaranteesJerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De SaNeurIPS 2023 · 503 citations
Related papers
- Effective Interplay between Sparsity and Quantization: From Theory to PracticeSimla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin et al.ICLR 2025
- Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian OptimizationYixin Ji, Yang Xiang, Juntao Li, Qingrong Xia et al.EMNLP Findings 2024 · 4 citations
- Compressing Large Language Models by Joint Sparsification and QuantizationJinyang Guo, Jianyu Wu, Zining Wang, Jiaheng Liu et al.ICML 2024 · 33 citations
- When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language ModelsWeilan Wang, Yu Mao, Dongdong Tang, Hongchao Du et al.EMNLP Findings 2024 · 7 citations
- MoDeGPT: Modular Decomposition for Large Language Model CompressionChi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel et al.ICLR 2025
