Lune

NeurIPS2025Top-tier venue

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training

Jintao Zhang, Jia Wei, Haoxu Wang, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Kai Jiang, Jianfei Chen, Jun Zhu

Abstract

The efficiency of attention is important due to its quadratic time complexity. We enhance the efficiency of attention through two key contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to accelerate attention computation. Our implementation achieves 1038 TOPS on RTX5090, which is a 5× speedup over the fastest FlashAttention on RTX5090. Experiments show that our FP4 attention can accelerate inference of various models in a plug-and-play way. Second, we pioneer low-bit attention to training tasks. Existing low-bit attention works like FlashAttention3 and SageAttention focus only on inference. However, the efficiency of training large models is also important. To explore whether low-bit attention can be effectively applied to training tasks, we design an accurate and efficient 8-bit attention for both forward and backward propagation. Experiments indicate that 8-bit attention achieves lossless performance in fine-tuning tasks but exhibits slower convergence in pretraining tasks. The code is available at https://github.com/thu-ml/SageAttention. 5x 212 1038 3x 490s 164s FlashAttention2 (End-to-End Time: 490s on RTX5090) SageAttention3 (End-to-End Time: 164s on RTX5090) Figure 1: The upper left figure shows the kernel speedup on RTX5090. The other two figures show the end-to-end inference speedup of generating a video using HunyuanVideo on RTX5090. Note that FlashAttention3 can only run on Hopper GPUs, so FlashAttention2 is already the fastest on RTX5090.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

28 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Radial Attention: 𝒪(n log n) Sparse Attention with Energy Decay for Long Video Generation cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • Radial Attention: 𝒪(n log n) Sparse Attention with Energy Decay for Long Video Generation cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Radial Attention: 𝒪(n log n) Sparse Attention with Energy Decay for Long Video Generation cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models cites R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
  • PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers28

  1. Radial Attention: 𝒪(n log n) Sparse Attention with Energy Decay for Long Video GenerationXingyang Li, Muyang Li, Tianle Cai, Haocheng Xi et al.NeurIPS 2025 · 66 citations
  2. R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token RoutingTianyu Fu, Yi Ge, Yichen You, Enshu Liu et al.NeurIPS 2025 · 32 citations
  3. QeRL: Beyond Efficiency - Quantization-enhanced Reinforcement Learning for LLMsWei Huang, Yi Ge, Shuai Yang, Yicheng Xiao et al.ICLR 2026 · 19 citations
  4. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation ModelsTianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao et al.NeurIPS 2025 · 19 citations
  5. FSA: An Alternative Efficient Implementation of Native Sparse Attention KernelRan Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai et al.ICLR 2026 · 10 citations

Builds on27

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  4. Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations
  5. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations

Related papers

  1. SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 QuantizationJintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei et al.ICML 2025
  2. Attn-QAT: 4-Bit Attention With Quantization-Aware TrainingPeiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang et al.ICML 2026 · 2 citations
  3. SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference AccelerationJintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu et al.ICLR 2025
  4. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionJay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar et al.NeurIPS 2024 · 727 citations
  5. BinaryAttention: One-Bit QK-Attention for Vision and Diffusion TransformersChaodong XIAO, Zhengqiang ZHANG, Lei ZhangCVPR 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines