EMNLP Findings2025
CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression
Dayin Gou, Sanghyun Byun, Nilesh Malpeddi, Gabrielle De Micheli, Prathamesh Vaste, Jacob Song, Woo Seong Chung
Abstract
Large Language Models (LLMs) typically rely on a large number of parameters for token embedding, leading to substantial storage requirements and memory footprints.In particular, LLMs deployed on edge devices are memory-bound, and reducing the memory footprint by compressing the embedding layer not only frees up the memory bandwidth but also speeds up inference.To address this, we introduce CARVQ, a post-training novel Corrective Adaptor combined with group Residual Vector Quantization.CARVQ relies on the composition of both linear and non-linear maps and mimics the original model embedding to compress to approximately 1.6 bits without requiring specialized hardware to support lower-bit storage.We test our method on pre-trained LLMs such as LLaMA-3.2-1B,LLaMA-3.2-3B,LLaMA-3.2-3B-Instruct,LLaMA-3.1-8B,Qwen2.5-7B,Qwen2.5-Math-7B and Phi-4, evaluating on common generative, discriminative, math and reasoning tasks.We show that in most cases, CARVQ can achieve lower average bitwidth-per-parameter while maintaining reasonable perplexity and accuracy compared to scalar quantization.Our contributions include a novel compression technique that is compatible with state-of-the-art transformer quantization methods and can be seamlessly integrated into any hardware supporting 4-bit memory to reduce the model's memory footprint in memoryconstrained devices.This work demonstrates a crucial step toward the efficient deployment of LLMs on edge devices.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture cites PIQA: Reasoning about Physical Commonsense in Natural Language
- EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Cited by top-tier papers1
Builds on12
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
Related papers
- SCVQ: Sparse-Compensated Vector Quantization for Large Language ModelsZixuan Zhou, Yujun Diao, Zicheng Kong, Dehua Ma et al.ACL 2026
- Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language ModelsHe Xiao, Qingyao Yang, Dirui Xie, Wendong Xu et al.ACL Findings 2026
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight CompressionTim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev et al.ICLR 2024 · 392 citations
- Compressing Large Language Models using Low Rank and Low Precision DecompositionRajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith et al.NeurIPS 2024 · 74 citations
- DecDEC: A Systems Approach to Advancing Low-Bit LLM QuantizationYeonhong Park, Jake Hyun, Hojoon Kim, Jae W. LeeOSDI 2025 · 9 citations
