ACL Findings2024
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, Chun Yuan
Abstract
Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outliers in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions with no extra inference overhead. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further with minimal training costs. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement over various quantization methods across different LLMs and downstream tasks, leading to the new state-of-the-art for LLM quantization. The codes are available at https://github.com/ruikangliu/IntactKV.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
18 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Efficient Streaming Language Models with Attention Sinks cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Efficient Streaming Language Models with Attention Sinks cites Language Models are Few-Shot Learners
- Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- PTQ4DiT: Post-training Quantization for Diffusion Transformers cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Efficient Streaming Language Models with Attention Sinks
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models cites Measuring Massive Multitask Language Understanding
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models cites Language Models are Few-Shot Learners
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs cites Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs cites PTQ4DiT: Post-training Quantization for Diffusion Transformers
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs cites Measuring Massive Multitask Language Understanding
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs cites Language Models are Few-Shot Learners
- PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs cites Efficient Streaming Language Models with Attention Sinks
- Fast Inference from Transformers via Speculative Decoding cites Language Models are Few-Shot Learners
Cited by top-tier papers18
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMsHaokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui et al.NeurIPS 2024 · 206 citations
- PTQ4DiT: Post-training Quantization for Diffusion TransformersJunyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah et al.NeurIPS 2024 · 87 citations
- Plug-and-Play: An Efficient Post-training Pruning Method for Large Language ModelsYingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao et al.ICLR 2024 · 72 citations
- PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMsTengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao et al.ICLR 2026 · 18 citations
Builds on16
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
Related papers
- Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache CompressionPeiyu Liu, Ze-Feng Gao, Xin Zhao, Yipeng Ma et al.ACL 2024 · 2 citations
- Accurate KV Cache Quantization with Outlier Tokens TracingYi Su, Yuechi Zhou, Quantong Qiu, Juntao Li et al.ACL 2025
- MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtizationZongwu Wang, Peng Xu, Fangxin Liu, Yiwei Hu et al.DAC 2025 · 6 citations
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsSaleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li et al.NeurIPS 2024 · 723 citations
- More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache CompressionJiebin Zhang, Dawei Zhu, Yifan Song, Wenhao Wu et al.EMNLP Findings 2025 · 2 citations
