Lune

ACL Findings2025

RQT: Hierarchical Residual Quantization for Multi-Model Compression

Tianqi Chen, Peisong Wang, Weixiang Xu, Zeyu Zhu, Jian Cheng

Abstract

Delta compression methods focus on efficiently serving multiple uniquely fine-tuned models, each tailored to specific tasks and user requirements. These approaches decompose a finetuned LLM into a base model and corresponding delta weights, which are compressed using low-rank or low-bit representations to reduce storage costs. However, their effectiveness is highly sensitive to the magnitude of the model deltas-a factor directly influenced by the scale of the training data. We propose the Residual Quantization Tree (RQT), a hierarchical quantization framework that automatically shares low-bit integer weights across similar fine-tuned models. The RQT construction employs a two-phase greedy algorithm: a bottom-up aggregation of models based on weight matrix similarity and top-down residual quantization, in which each node optimizes the quantization parameters and then delegates residual errors to child nodes. We evaluate RQT on fine-tuned models across mathematics, coding, chatbot, and Chinese LLMs. The results show that RQT achieves an average accuracy degradation of approximately 3% (comparable to previous 4-bit post-training quantization) while maintaining an effective bitwidth of around 2 bits.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SGLang: Efficient Execution of Structured Language Model Programs cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on19

  1. SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun et al.NeurIPS 2024 · 1,586 citations
  2. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
  5. Up or Down? Adaptive Rounding for Post-Training QuantizationMarkus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos et al.ICML 2020 · 816 citations

Related papers

  1. D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank ApproximationJunlin Li, Shuangyong Song, Guodong Du, Ngai Wong et al.ACL Findings 2026
  2. Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language ModelsBowen Ping, Shuo Wang, Hanqing Wang, Xu Han et al.NeurIPS 2024 · 25 citations
  3. MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight QuantizationChun Hu, Junhui He, Shangyu Wu, Yuxin He et al.EMNLP 2025 · 1 citation
  4. BitDelta: Your Fine-Tune May Only Be Worth One BitJames Liu, Guangxuan Xiao, Kai Li, Jason D. Lee et al.NeurIPS 2024 · 50 citations
  5. Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta CompressionXiaohui Wang, Peng Ye, Chenyu Huang, Shenghe Zheng et al.NeurIPS 2025
Dusk over the sea between two cliffs drawn in fine vertical lines