Lune

HPCA2025Top-tier venue

BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

Yuzong Chen, Ahmed F. AbouElhamayed, Xilai Dai, Yang Wang, Marta Andronic, George A. Constantinides, Mohamed S. Abdelfattah

Abstract

Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD 1 , an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bitserial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with < 0.5% accuracy loss on average. For generative tasks, BitMoD is able to quantize LLM weights to 3-bit while achieving better perplexity than prior LLM quantization scheme. Combining the superior model performance with an efficient accelerator design, BitMoD achieves an average of 1.69× and 1.48× speedups compared to prior LLM accelerators ANT and OliVe, respectively.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion cites Language Models are Few-Shot Learners
  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • QLoRA: Efficient Finetuning of Quantized LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models cites Language Models are Few-Shot Learners
  • GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • \mathrm{P}^{3}-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • \mathrm{P}^{3}-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats cites GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference
  • Language Models are Few-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization cites QLoRA: Efficient Finetuning of Quantized LLMs
  • M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. P3\mathrm{P}^{3}-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical FormatsYuzong Chen, Chao Fang, Xilai Dai, Yuheng Wu et al.ISCA 2026 · 4 citations
  2. PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage FusionHuizheng Wang, Hongbin Wang, Zichuan Wang, Zhiheng Yue et al.HPCA 2026 · 2 citations
  3. GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM InferenceSangjin Kim, Yuseon Chou, Byeongcheol Kim, Jungjun Oh et al.HPCA 2026 · 2 citations
  4. M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit QuantizationWeiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang et al.ASPLOS 2026 · 2 citations
  5. Focus: A Streaming Concentration Architecture for Efficient Vision-Language ModelsChiyue Wei, Cong Guo, Junyao Zhang, Haoxuan Shan et al.HPCA 2026 · 2 citations

Builds on27

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  3. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  4. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  5. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations

Related papers

  1. An Algorithm-Hardware Co-design Based on Revised Microscaling Format Quantization for Accelerating Large Language ModelsYingbo Hao, Huangxu Chen, Yi Zou, Yanfeng YangDAC 2025 · 1 citation
  2. Amove: Accelerating LLMs through Mitigating Outliers and Salient Points via Fine-Grained Grouped Vectorized Data TypeXilong Xie, Liang Wang, Limin Xiao, Meng Han et al.MICRO 2025 · 2 citations
  3. Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data FormatChao Fang, Man Shi, Robin Geens, Arne Symons et al.HPCA 2025 · 15 citations
  4. S-Quant: Rethinking Weight Quantization with Seed-Based GenerationMingzi Wang, Lancheng Zou, Shuo Yin, Zhuolun He et al.ICML 2026
  5. OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language ModelsJahyun Koo, Dahoon Park, Sangwoo Jung, Jaeha KungDAC 2024 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines