A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
Hongsun Jang, Jaeyong Song, Changmin Shin, Si Ung Noh, Jaewon Jung, Jisung Park, Jinho Lee
Abstract
The computational and memory demands of large language models for generative inference present significant challenges for practical deployment. One promising solution targeting offline inference is offloading-based batched inference, which extends the GPU's memory hierarchy with host memory and storage. However, it often suffers from substantial I/O overhead, primarily due to the large KV cache sizes that scale with batch size and context window length.
In this paper, we introduce HILOS, a framework that boosts offline inference throughput using near-storage processing. The core of HILOS is attention near storage, which offloads memory-intensive attention operations to near-storage accelerators, reducing traffic across the system interconnect. Building on attention near storage, HILOS incorporates three additional optimizations. First, cooperative X-cache minimizes KV cache I/O by exploiting available host resources after offloading. Second, delayed KV cache writeback hides storage write latency and mitigates storage write amplification. Finally, a memory-efficient attention accelerator sustains high throughput for long sequences within the resource constraints of NSP devices. We implemented and evaluated HI-LOS on a real system equipped with 16 SmartSSDs. Compared to state-of-the-art offloading-based inference frameworks, HILOS achieves up to 7.86× throughput while reducing energy consumption by up to 85%. The source code for HI-LOS is available at https://github.com/hongsunjang/HILOS.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Language Models are Few-Shot Learners
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Efficient Memory Management for Large Language Model Serving with PagedAttention cites Language Models are Few-Shot Learners
Cited by top-tier papers1
Builds on64
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
Related papers
- InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM InferenceXiurui Pan, Endian Li, Qiao Li, Shengwen Liang et al.HPCA 2025 · 22 citations
- Scaling Attention Beyond GPUs for LLM InferenceWeishu Deng, Yujie Yang, Peiran Du, Lingfeng Xiang et al.HPDC 2026
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM InferenceHanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng et al.ICML 2025
- Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real SystemHongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park et al.HPCA 2024 · 26 citations
- LiLo: Harnessing the on-Chip Accelerators in Intel CPUs for Compressed LLM Inference AccelerationHyungyo Kim, Qirong Xia, Jinghan Huang, Nachuan Wang et al.HPCA 2026 · 1 citation
