Lune

ISCA2024Top-tier venue

Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

Jungi Lee, Wonbeom Lee, Jaewoong Sim

Abstract

Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning and have thus become one of the most important workloads in today’s computing landscape. However, deploying LLM inference poses challenges due to the high compute and memory requirements stemming from the enormous model size and the difficulty of running it in the integer pipelines. In this paper, we present Tender, an algorithm-hardware co-design solution that enables efficient deployment of LLM inference at low precision. Based on our analysis of outlier values in LLMs, we propose a decomposed quantization technique in which the scale factors of decomposed matrices are powers of two apart. The proposed scheme allows us to avoid explicit requantization (i.e., dequantization/quantization) when accumulating the partial sums from the decomposed matrices, with a minimal extension to the commodity tensor compute hardware. Our evaluation shows that Tender achieves higher accuracy and inference performance compared to the state-of-the-art methods while also being significantly less intrusive to the existing accelerators.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

19 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • Reassessing Layer Pruning in LLMs: New Insights and Methods cites QLoRA: Efficient Finetuning of Quantized LLMs
  • QLoRA: Efficient Finetuning of Quantized LLMs cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • QLoRA: Efficient Finetuning of Quantized LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization cites Orca: A Distributed Serving System for Transformer-Based Generative Models
  • M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type cites QLoRA: Efficient Finetuning of Quantized LLMs
  • M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management cites Orca: A Distributed Serving System for Transformer-Based Generative Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers19

  1. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache ManagementWonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong SimOSDI 2024 · 248 citations
  2. Reassessing Layer Pruning in LLMs: New Insights and MethodsYao Lu, Hao Cheng, Yujie Fang, Zeyu Wang et al.ICLR 2026 · 24 citations
  3. M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical TypeWeiming Hu, Haoyan Zhang, Cong Guo, Yu Feng et al.HPCA 2025 · 19 citations
  4. PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMsJiajun Qin, Tianhua Xia, Cheng Tan, Jeff Zhang et al.ASPLOS 2025 · 17 citations
  5. Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache QuantizationMinsu Kim, Seongmin Hong, Ryeowook Ko, Soongyu Choi et al.ISCA 2025 · 17 citations

Builds on26

  1. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  2. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
  3. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersZhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu et al.NeurIPS 2022 · 816 citations
  4. Orca: A Distributed Serving System for Transformer-Based Generative ModelsGyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim et al.OSDI 2022 · 690 citations
  5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li et al.ICML 2023 · 683 citations

Related papers

  1. Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache CompressionPeiyu Liu, Ze-Feng Gao, Xin Zhao, Yipeng Ma et al.ACL 2024 · 2 citations
  2. DuoQ: A DSP Utilization-aware and Outlier-free Quantization for FPGA-based LLMs AccelerationZhuoquan Yu, Huidong Ji, Yue Cao, Junfu Wu et al.DAC 2025 · 1 citation
  3. BitMoD: Bit-serial Mixture-of-Datatype LLM AccelerationYuzong Chen, Ahmed F. AbouElhamayed, Xilai Dai, Yang Wang et al.HPCA 2025 · 23 citations
  4. TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM QuantizationHaodong WANG, Junjie Liu, Zicong Hong, Qianli Liu et al.ICML 2026
  5. ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language ModelsChao Zeng, Songwei Liu, Yusheng Xie, Hong Liu et al.AAAI 2025 · 24 citations
Dusk over the sea between two cliffs drawn in fine vertical lines