Lune

ICLR2026Top-tier venue

DynamicInfer: Runtime-Aware Sparse Offloading for LLMs Inference on a Consumer-Grade GPU

Zhui Zhu, Weichen Zhang, Zhenghan Zhou, Yunhao Liu, Fan Dang

Abstract

Large Language Models (LLMs) have achieved remarkable success in various NLP tasks, but their enormous memory footprints pose significant challenges for deployment on consumer-grade GPUs. Prior solutions, such as PowerInfer, combine offloading and sparse activation to reduce memory and computational overhead, but suffer from static neuron partitioning, leading to suboptimal GPU utilization and increased latency. In this work, we present DynamicInfer, a runtime neuron offloading framework that dynamically adapts neuron scheduling based on input-dependent activation patterns. DynamicInfer introduces (1) a hierarchical neural caching strategies, (2) a load-aware neuron activation mechanism tailored to heterogeneous hardware, and (3) an activation-aware prefetching pipeline that overlaps data transfer with computation. Extensive experiments on ReluLLaMA and Prosparse models across multiple hardware platforms demonstrate that Dy-namicInfer achieves up to 253% speedup over llama.cpp and 59% over PowerInfer, while retaining model accuracy. Our approach offers a practical and scalable solution for high-performance LLM inference on resource-constrained devices. INTRODUCTION Large language models (LLMs) have revolutionized natural language processing tasks such as text summarization (

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on21

  1. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
  5. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations

Related papers

  1. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUYixin Song, Zeyu Mi, Haotong Xie, Haibo ChenSOSP 2024 · 86 citations
  2. Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMMLian Liu, Shixin Zhao, Bing Li, Haimeng Ren et al.HPCA 2025 · 15 citations
  3. Kairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron BalancingYapeng Jiang, Minghao Gan, Zicong Hong, Wuhui Chen et al.OSDI 2026
  4. SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token PruningLingkun Long, Rubing Yang, Yushi Huang, Desheng Hui et al.AAAI 2026 · 8 citations
  5. LLM in a flash: Efficient Large Language Model Inference with Limited MemoryKeivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, S. Khatamifard et al.ACL 2024 · 73 citations
Dusk over the sea between two cliffs drawn in fine vertical lines