FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao
Abstract
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2 achieving only 35% utilization on the H100 GPU. We develop three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) block quantization and incoherent processing that leverages hardware support for FP8 low-precision. We demonstrate that our method, FlashAttention-3, achieves speedup on H100 GPUs by 1.5-2.0 with FP16 reaching up to 740 TFLOPs/s (75% utilization), and with FP8 reaching close to 1.2 PFLOPs/s. We validate that FP8 FlashAttention-3 achieves 2.6 lower numerical error than a baseline FP8 attention.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
192 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention cites Reformer: The Efficient Transformer
- Big Bird: Transformers for Longer Sequences cites Reformer: The Efficient Transformer
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Big Bird: Transformers for Longer Sequences
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Reformer: The Efficient Transformer
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion cites Video Diffusion Models
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation cites Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
Cited by top-tier papers192
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video DiffusionXun Huang, Zhengqi Li, Guande He, Mingyuan Zhou et al.NeurIPS 2025 · 628 citations
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and InferenceBenjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller et al.ACL 2025 · 552 citations
- TurboQuant: Online Vector Quantization with Near-optimal Distortion RateAmir Zandieh, Majid Daliri, Majid Hadian, Vahab MirrokniICLR 2026 · 132 citations
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware PermutationShuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li et al.NeurIPS 2025 · 114 citations
- Autoregressive Adversarial Post-Training for Real-Time Interactive Video GenerationShanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang et al.NeurIPS 2025 · 89 citations
Builds on18
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
- Video Diffusion ModelsJonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan et al.NeurIPS 2022 · 2,948 citations
- Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
- Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations
Related papers
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector ParallelismJianxing Xu, Yuanbo Wen, Jun Bi, Ruibai Xu et al.PPoPP 2026
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 QuantizationJintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei et al.ICML 2025
- ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision TransformersChih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao, Chia-Ming LeeCVPR Findings 2026
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit TrainingJintao Zhang, Jia Wei, Haoxu Wang, Pengle Zhang et al.NeurIPS 2025 · 81 citations
