Lune

ASPLOS2026Top-tier venue

Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration

Zejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen, Yutong Lu, Xianwei Zhang

Abstract

Modern large language model (LLM) serving systems confront inefficient GPU utilization due to the fundamental mismatch between compute-intensive prefill phase and memory-bound decode phase. While current practices attempt to address this by organizing these phases into hybrid batches, such solutions create an inefficient tradeoff that sacrifices either throughput or latency, leaving substantial GPU resources underutilized. For this, we identify two key root causes: 1) the prefill phase suffers from suboptimal compute utilization due to wave quantization and attention bottlenecks, and 2) hybrid batching disproportionately prioritizes latency over throughput, wasting both compute resources and memory bandwidth. To mitigate the issues, we present Bullet, a novel spatial-temporal orchestration system that eliminates these inefficiencies through fine-grained phase coordination. Bullet enables concurrent execution of prefill and decode requests, while dynamically provisioning GPU resources based on real-time performance modeling. By integrating SLO-aware scheduling and adaptive resource allocation, Bullet maximizes GPU utilization without compromising latency targets. Experimental evaluations on real-world workloads demonstrate that Bullet delivers 1.26× average throughput gains (up to 1.55×) over state-of-the-arts, while consistently meeting latency constraints.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism cites SGLang: Efficient Execution of Structured Language Model Programs
  • DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing cites SGLang: Efficient Execution of Structured Language Model Programs
  • Beyond Utilization: Energy-Conscious GPU Sharing for Inference Serving cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • BlendServe: Optimizing Offline Inference with Resource-Aware Batching cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • BlendServe: Optimizing Offline Inference with Resource-Aware Batching cites SGLang: Efficient Execution of Structured Language Model Programs
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • SGLang: Efficient Execution of Structured Language Model Programs cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • SGLang: Efficient Execution of Structured Language Model Programs cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline ParallelismHongxin Xu, Tianyu Guo, Xianwei ZhangNeurIPS 2025 · 4 citations
  2. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU MultiplexingLei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong et al.ICML 2026 · 4 citations
  3. Beyond Utilization: Energy-Conscious GPU Sharing for Inference ServingPrasoon Sinha, Dimitrios Liakopoulos, Nathan Lemma, Neeraja J. YadwadkarSOSP 2026
  4. BlendServe: Optimizing Offline Inference with Resource-Aware BatchingYilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng et al.ASPLOS 2026
  5. MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block RepresentationsJiaxiang Zou, Yonghao Chen, Ruilong WU, Xinyu ChenICML 2026

Builds on35

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  4. SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun et al.NeurIPS 2024 · 1,586 citations
  5. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations

Related papers

  1. Towards High-Goodput LLM Serving with Prefill-decode MultiplexingYukang Chen, Weihao Cui, Han Zhao, Ziyi Xu et al.ASPLOS 2026 · 18 citations
  2. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model ServingYinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu et al.OSDI 2024 · 646 citations
  3. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM InferenceAditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter et al.ASPLOS 2025 · 24 citations
  4. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeAmey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan et al.OSDI 2024 · 537 citations
  5. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic SchedulingJingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang et al.ISCA 2025 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines