Lune

HPCA2025Top-tier venue

InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, Jie Zhang

Abstract

The widespread of Large Language Models (LLMs) marks a significant milestone in generative AI. Nevertheless, the increasing context length and batch size in offline LLM inference escalate the memory requirement of the key-value (KV) cache, which imposes a huge burden on the GPU VRAM, especially for resource-constrained scenarios (e.g., edge computing). Several cost-effective solutions leverage host memory or SSDs to reduce storage costs for offline inference scenarios and improve the throughput. Nevertheless, they suffer from significant performance penalties imposed by intensive KV cache accesses due to limited PCIe bandwidth. To address these issues, we propose InstAttention, a novel LLM inference system that offloads the most performance-critical computation (i.e., attention in decoding phase) and data (i.e., KV cache) parts to Computational Storage Drives (CSDs), which minimize the enormous KV transfer overheads. InstAttention designs a dedicated flashaware in-storage attention engine with KV cache management mechanisms to exploit the high internal bandwidths of CSDs instead of being limited by the PCIe bandwidth. The optimized P2P transmission between GPU and CSDs further reduces data migration overheads. Experimental results demonstrate that for a 13B model using an NVIDIA A6000 GPU, InstAttention improves throughput for long-sequence inference by up to 11.1×11.1 \times, compared to existing SSD-based solutions such as FlexGen.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

its abstract · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in Solid State Drives cites A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMQingyuan Liu, Liyan Chen, Haocheng Wang, Yanning Yang et al.ISCA 2026 · 6 citations
  2. Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in Solid State DrivesRakesh Nadig, Vamanan Arulchelvan, Mayank Kabra, Harshita Gupta et al.HPCA 2026 · 2 citations
  3. Experiential Fairness: Bridging the Gap Between User Experience and Resource-Centric Fairness in Online LLM ServicesJiahua Huang, Wentai Wu, Yongheng Liu, Guozhi Liu et al.AAAI 2026
  4. A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMsHongsun Jang, Jaeyong Song, Changmin Shin, Si Ung Noh et al.ASPLOS 2026

Related papers

  1. No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMsShutian Luo, Haiying ShenOSDI 2026
  2. SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCsXinrui Zheng, Dongliang Wei, Jianxiang Gao, Yixin Song et al.FAST 2026 · 9 citations
  3. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM InferenceHanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng et al.ICML 2025
  4. PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer InferencesJie Ye, Avinash Maurya, Krishna Teja Chitty-Venkata, Bogdan Nicolae et al.HPDC 2026
  5. Accelerating LLM Inference Throughput via Asynchronous KV Cache PrefetchingYanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng et al.AAAI 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines