Lune

HPCA2026Top-tier venue

AQPIM: Breaking the PIM Capacity Wall for LLMs with in-Memory Activation Quantization

Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura, Daichi Fujiki

Abstract

Processing-in-Memory (PIM) architectures offer a promising solution to the memory bottlenecks in data-intensive machine learning, yet often overlook the growing challenge of activation memory footprint. Conventional PIM approaches struggle with massive KV cache sizes generated in long-context scenarios by Transformer-based models, frequently exceeding PIM's limited memory capacity, while techniques like sparse attention can conflict with PIM's need for data locality. Existing PIM approaches and quantization methods are often insufficient or poorly suited for leveraging the unique characteristics of activations. This work identifies an opportunity for PIMspecialized activation quantization to enhance bandwidth and compute efficiency. We explore clustering-based vector quantization approaches, which align well with activation characteristics and PIM's internal bandwidth capabilities. Building on this, we introduce AQPIM, a novel PIM-aware activation quantization framework based on Product Quantization (PQ), optimizing it for modern Large Language Models (LLMs). By performing quantization directly within memory, AQPIM leverages PIM's high internal bandwidth and enables direct computation on compressed data, significantly reducing both memory footprint and computational overhead for attention computation. AQPIM addresses PQ's accuracy challenges by introducing several algorithmic optimizations. Evaluations demonstrate that AQPIM achieves significant performance improvements, drastically reducing of GPU-CPU communication that can account for90∼98.5%90 \sim 98.5 \%of decoding latency, together with3.4×3.4 \timesspeedup over a SOTA PIM approach.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Streaming Language Models with Attention Sinks cites Big Bird: Transformers for Longer Sequences
  • Efficient Streaming Language Models with Attention Sinks cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Efficient Streaming Language Models with Attention Sinks cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Big Bird: Transformers for Longer Sequences
  • \mathrm{P}^{3}-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. P3\mathrm{P}^{3}-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical FormatsYuzong Chen, Chao Fang, Xilai Dai, Yuheng Wu et al.ISCA 2026 · 4 citations

Builds on33

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  3. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  4. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  5. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations

Related papers

  1. MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtizationZongwu Wang, Peng Xu, Fangxin Liu, Yiwei Hu et al.DAC 2025 · 6 citations
  2. VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM InferenceZihan Liu, Xinhao Luo, Junxian Guo, Wentao Ni et al.HPCA 2025 · 11 citations
  3. OutlierCIM: Outlier-Aware Digital CIM-Based LLM Accelerator with Hybrid-Strategy Quantization and Unified FP-INT ComputationZihan Zou, Shikuang Chen, Chen Zhang, Xing Wang et al.DAC 2025
  4. STARC: Selective Token Access with Remapping and Clustering for Efficient LLM Decoding on PIM SystemsZehao Fan, Yunzhen Liu, Garrett Gagnon, Zhenyu Liu et al.ASPLOS 2026
  5. COMET: Towards Practical W4A4KV4 LLMs ServingLian Liu, Long Cheng, Haimeng Ren, Zhaohui Xu et al.ASPLOS 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines