Lune

EMNLP Findings2020

Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, Dongsoo Lee

Abstract

The deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead during inference, especially when the target device is limited in computational resources such as mobile or edge devices. Quantization is an effective technique to address such challenges. Our analysis shows that for a given number of quantization bits, each block of Transformer contributes to translation quality and inference computations in different manners. Moreover, even inside an embedding block, each word presents vastly different contributions. Correspondingly, we propose a mixed precision quantization strategy to represent Transformer weights by an extremely low number of bits (e.g., under 3 bits). For example, for each word in an embedding block, we assign different quantization bits based on statistical property. Our quantized Transformer model achieves 11.8× smaller model size than the baseline model, with less than -0.5 BLEU. We achieve 8.3× reduction in run-time memory footprints and 3.5× speed up (Galaxy N10+) such that our proposed compression strategy enables efficient implementation for on-device NMT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized DNNs cites And the Bit Goes Down: Revisiting the Quantization of Neural Networks
  • BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized DNNs cites Fully Quantized Transformer for Machine Translation
  • FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables cites BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized DNNs
  • FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables cites LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
  • FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables cites SqueezeLLM: Dense-and-Sparse Quantization
  • LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models cites BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized DNNs
  • LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models cites SqueezeLLM: Dense-and-Sparse Quantization
  • Understanding Int4 Quantization for Language Models: Latency Speedup, Composability, and Failure Cases cites HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. SqueezeLLM: Dense-and-Sparse QuantizationSehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong et al.ICML 2024 · 306 citations
  2. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language ModelsGunho Park, Baeseong Park, Minsub Kim, Sungjae Lee et al.ICLR 2024 · 134 citations
  3. Understanding Int4 Quantization for Language Models: Latency Speedup, Composability, and Failure CasesXiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao et al.ICML 2023 · 41 citations
  4. Mokey: enabling narrow fixed-point inference for out-of-the-box floating-point transformer modelsAli Hadi Zadeh, Mostafa Mahmoud, Ameer Abdelhadi, Andreas MoshovosISCA 2022 · 36 citations
  5. FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up TablesGunho Park, Hyeokjun Kwon, Jiwoo Kim, Jeongin Bae et al.HPCA 2025 · 10 citations

Builds on4

  1. HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-PrecisionZhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney et al.ICCV 2019 · 645 citations
  2. And the Bit Goes Down: Revisiting the Quantization of Neural NetworksPierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham et al.ICLR 2020 · 157 citations
  3. Fully Quantized Transformer for Machine TranslationGabriele Prato, Ella Charlaix, Mehdi RezagholizadehEMNLP Findings 2020 · 62 citations
  4. BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized DNNsYongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim et al.SC 2020 · 31 citations

Related papers

  1. Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine TranslationSunzhu Li, Peng Zhang, Guobing Gan, Xiuqing Lv et al.EMNLP 2022 · 3 citations
  2. CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding CompressionDayin Gou, Sanghyun Byun, Nilesh Malpeddi, Gabrielle De Micheli et al.EMNLP Findings 2025 · 1 citation
  3. Lite Transformer with Long-Short Range AttentionZhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin et al.ICLR 2020 · 379 citations
  4. MobileQuant: Mobile-friendly Quantization for On-device Language ModelsFuwen Tan, Royson Lee, Lukasz Dudziak, Shell Xu Hu et al.EMNLP Findings 2024 · 13 citations
  5. 8-bit Transformer Inference and Fine-tuning for Edge AcceleratorsJeffrey Yu, Kartik Prabhu, Yonatan Urman, Robert M. Radway et al.ASPLOS 2024 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines