LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Abstract
The deployment of large language models (LLMs) is frequently hindered by prohibitive memory and computational requirements. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub-1-bit regime remains a persistent challenge. In this paper, we introduce LittleBit, a novel framework for extreme LLM compression. We target quantization rates as low as bits per weight (BPW), achieving a memory reduction of approximately , which effectively compresses Llama2-13B to under GB. We represent weights via low-rank latent matrix factorization and subsequently binarize the resulting factors. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions. Two primary contributions enable effective training: Dual Sign-Value-Independent Decomposition (Dual-SVID) for quantization-aware training (QAT) initialization, and Residual Compensation to minimize approximation errors. Extensive experiments confirm the superiority of LittleBit in the sub-1-bit domain; for instance, our method at BPW surpasses the performance of leading techniques operating at BPW on Llama2-7B. We establish a new size-performance trade-off -- unlocking a potential inference speedup relative to FP16 -- and render powerful LLMs practical for resource-constrained environments. Our code is available at https://github.com/SamsungLabs/LittleBit.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
- A Simple and Effective Pruning Approach for Large Language Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization cites OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization cites PIQA: Reasoning about Physical Commonsense in Natural Language
- NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models cites OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
- NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models cites Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
Cited by top-tier papers2
Builds on21
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
- Efficient large-scale language model training on GPU clusters using megatron-LMDeepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley et al.SC 2021 · 576 citations
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language ModelsWenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu et al.ICLR 2024 · 395 citations
Related papers
- LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry AlignmentBanseok Lee, Youngmin KimICML 2026
- OneBit: Towards Extremely Low-bit Large Language ModelsYuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang et al.NeurIPS 2024 · 110 citations
- SCVQ: Sparse-Compensated Vector Quantization for Large Language ModelsZixuan Zhou, Yujun Diao, Zicheng Kong, Dehua Ma et al.ACL 2026
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMsWei Huang, Yangdong Liu, Haotong Qin, Ying Li et al.ICML 2024 · 161 citations
- PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language ModelsJiaqi Zhao, Miao Zhang, Ming Wang, Yuzhang Shang et al.ACL 2025 · 7 citations
