Lune

ACL Findings2025

RQT: Hierarchical Residual Quantization for Multi-Model Compression

Tianqi Chen, Peisong Wang, Weixiang Xu, Zeyu Zhu, Jian Cheng

摘要

Delta compression methods focus on efficiently serving multiple uniquely fine-tuned models, each tailored to specific tasks and user requirements. These approaches decompose a finetuned LLM into a base model and corresponding delta weights, which are compressed using low-rank or low-bit representations to reduce storage costs. However, their effectiveness is highly sensitive to the magnitude of the model deltas-a factor directly influenced by the scale of the training data. We propose the Residual Quantization Tree (RQT), a hierarchical quantization framework that automatically shares low-bit integer weights across similar fine-tuned models. The RQT construction employs a two-phase greedy algorithm: a bottom-up aggregation of models based on weight matrix similarity and top-down residual quantization, in which each node optimizes the quantization parameters and then delegates residual errors to child nodes. We evaluate RQT on fine-tuned models across mathematics, coding, chatbot, and Chinese LLMs. The results show that RQT achieves an average accuracy degradation of approximately 3% (comparable to previous 4-bit post-training quantization) while maintaining an effective bitwidth of around 2 bits.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • SGLang: Efficient Execution of Structured Language Model Programs 引用了 Efficient Memory Management for Large Language Model Serving with PagedAttention
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper19

  1. SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun 等NeurIPS 2024 · 1,586 次引用
  2. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng 等ICLR 2024 · 1,206 次引用
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng 等SOSP 2023 · 1,016 次引用
  4. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun 等ICLR 2024 · 945 次引用
  5. Up or Down? Adaptive Rounding for Post-Training QuantizationMarkus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos 等ICML 2020 · 816 次引用

相关 Paper

  1. D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank ApproximationJunlin Li, Shuangyong Song, Guodong Du, Ngai Wong 等ACL Findings 2026
  2. Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language ModelsBowen Ping, Shuo Wang, Hanqing Wang, Xu Han 等NeurIPS 2024 · 25 次引用
  3. MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight QuantizationChun Hu, Junhui He, Shangyu Wu, Yuxin He 等EMNLP 2025 · 1 次引用
  4. BitDelta: Your Fine-Tune May Only Be Worth One BitJames Liu, Guangxuan Xiao, Kai Li, Jason D. Lee 等NeurIPS 2024 · 50 次引用
  5. Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta CompressionXiaohui Wang, Peng Ye, Chenyu Huang, Shenghe Zheng 等NeurIPS 2025
黄昏的海面,两侧是细线勾勒的悬崖