MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtization
Zongwu Wang, Peng Xu, Fangxin Liu, Yiwei Hu, Qingxiao Sun, Gezi Li, Cheng Li, Xuan Wang, Li Jiang, Haibing Guan
Abstract
Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128 K or 1 M tokens. This trend, however, presents significant challenges in inference speed and memory management. The primary bottleneck in long-context LLM inference is the quadratic computational complexity of attention mechanisms, causing substantial slowdowns as sequence length increases. KV cache mechanism alleviates this issue by storing pre-computed data, but introduces memory requirements that scale linearly with context length, hindering efficient LLM deployment. Quantization emerges as a promising approach to address the widening gap between LLM size and memory capacity. However, traditional quantization schemes often yield suboptimal compression results for KV caches due to two key factors: i) On-the-fly quantization and de-quantization, causing significant performance overhead; ii) Prevalence of outliers in KV values, challenging low-bitwidth uniform quantization. To this end, we propose MILLION, a novel quantization framework achieving low-bitwidth KV cache through product quantization. First, we conduct a thorough analysis of KV cache distribution, revealing the limitations of existing quantization schemes. Second, we introduce a non-uniform quantization algorithm based on product quantization, which efficiently compresses data while preserving accuracy. Third, we develop a high-performance GPU inference framework with efficient attention kernel and pipeline design for MILLION that leverages sparse computation and asynchronous quantization, significantly enhancing inference speed. Comprehensive evaluation results demonstrate that MILLION can achieve 4 bits quantization with trivial perplexity and accuracy loss, and achieve 2.09 x end-to-end performance gains at 32 K context length. Code is released at https://github.com/ZongwuWang/MILLION.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Efficient Streaming Language Models with Attention Sinks cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Efficient Streaming Language Models with Attention Sinks
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Efficient Streaming Language Models with Attention Sinks
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
- TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference cites KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference cites SnapKV: LLM Knows What You are Looking for Before Generation
- SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
- SnapKV: LLM Knows What You are Looking for Before Generation cites H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization cites KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization cites QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
Cited by top-tier papers2
- VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector QuantizationDingyu Yao, Chenxu Yang, Zhengyang Tong, Zheng Lin et al.ACL 2026 · 4 citations
- TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode InferenceXiaojuan Tang, Fanxu Meng, Pingzhi Tang, Yuxuan Wang et al.ASPLOS 2026
Builds on17
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsSaleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li et al.NeurIPS 2024 · 723 citations
Related papers
- JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context InferenceChengyu Sun, Yaqi Xia, Hulin Wang, Donglin Yang et al.PPoPP 2026 · 1 citation
- PQCache: Product Quantization-based KVCache for Long Context LLM InferenceHailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu et al.SIGMOD 2025 · 13 citations
- KVO-LLM: Boosting Long-Context Generation Throughput for Batched LLM InferenceZhenyu Li, Dongxu Lyu, Gang Wang, Yuzhou Chen et al.DAC 2025 · 1 citation
- MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context InferenceAkshat Sharma, Hangliang Ding, Jianping Li, Neel Dani et al.ACL Findings 2025 · 1 citation
- ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMsYanlin Qi, Xinhang Chen, Huiqiang Jiang, Qitong Wang et al.ICML 2026
