POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, Ashish Panwar
Abstract
Each request in LLM inference goes through two phases: compute-bound prefill and memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid batching that combines the prefill and decode phases of different requests into the same batch. This approach optimizes linear operations but remains inefficient for attention computation because existing attention kernels specialize execution independently for the prefill and decode phases. In this paper, we present POD-Attention - the first GPU kernel that efficiently computes attention for hybrid batches. POD-Attention aims to maximize the utilization of both compute and memory bandwidth by carefully allocating the GPU's resources such that prefill and decode operations happen concurrently on the same multiprocessor. POD-Attention speeds up attention computation by up to (mean ), enabling higher throughput and lower latency LLM inference compared to the use of independently optimized prefill and decode attention kernels.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
21 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tail-Optimized Caching for LLM Inference cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Tail-Optimized Caching for LLM Inference cites SGLang: Efficient Execution of Structured Language Model Programs
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites Resource Multiplexing in Tuning and Serving Large Language Models
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites SGLang: Efficient Execution of Structured Language Model Programs
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective cites vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Resource Multiplexing in Tuning and Serving Large Language Models cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Towards High-Goodput LLM Serving with Prefill-decode Multiplexing cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Towards High-Goodput LLM Serving with Prefill-decode Multiplexing cites SGLang: Efficient Execution of Structured Language Model Programs
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- SGLang: Efficient Execution of Structured Language Model Programs cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- SGLang: Efficient Execution of Structured Language Model Programs cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention cites FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention cites SGLang: Efficient Execution of Structured Language Model Programs
Cited by top-tier papers21
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentionRamya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee et al.ASPLOS 2025 · 38 citations
- Towards High-Goodput LLM Serving with Prefill-decode MultiplexingYukang Chen, Weihao Cui, Han Zhao, Ziyi Xu et al.ASPLOS 2026 · 18 citations
- Efficient LLM Serving for Agentic Workflows: A Data Systems PerspectiveNoppanat Wadlom, Junyi Shen, Yao LuSIGMOD 2026 · 14 citations
- Resource Multiplexing in Tuning and Serving Large Language ModelsYongjun He, Haofeng Yang, Yao Lu, Ana Klimovic et al.USENIX ATC 2025 · 9 citations
- Tail-Optimized Caching for LLM InferenceWenxin Zhang, Yueying Li, Ciamac C. Moallemi, Tianyi PengNeurIPS 2025 · 6 citations
Builds on19
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun et al.NeurIPS 2024 · 1,586 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionJay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar et al.NeurIPS 2024 · 727 citations
Related papers
- Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal OrchestrationZejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen et al.ASPLOS 2026 · 2 citations
- PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile KernelJinjun Yi, Zhixin Zhao, Yitao Hu, Ke Yan et al.ASPLOS 2026 · 1 citation
- Hybe: GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context WindowSeungjae Moon, Junseo Cha, Hyunjun Park, Joo-Young KimISCA 2025 · 11 citations
- DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU MultiplexingLei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong et al.ICML 2026 · 4 citations
- Accelerating LLM Inference Throughput via Asynchronous KV Cache PrefetchingYanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng et al.AAAI 2026 · 4 citations
