Lune

ACL Findings2026

Revealing the Attention Floating Mechanism in Masked Diffusion Models

Xin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, Maosong Sun

Abstract

Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong incontext learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https:// github.com/NEUIR/Attention-Floating.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Diffusion Models cites Diffusion-LM Improves Controllable Text Generation
  • Large Language Diffusion Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding cites Large Language Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on16

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  3. Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations
  4. Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
  5. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel DecodingChengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu et al.ICLR 2026 · 428 citations

Related papers

  1. Attention Sinks in Diffusion Language ModelsMaximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri et al.ACL Findings 2026
  2. Masks Can Be Distracting: On Context Comprehension in Diffusion Language ModelsJulianna Piskorz, Cristina Pinneri, Alvaro Correia, Motasem Alfarra et al.ICML 2026 · 5 citations
  3. ReFusion: A Diffusion Large Language Model with Parallel Autoregressive DecodingJia-Nan Li, Jian Guan, Wei Wu, Chongxuan LiICLR 2026 · 17 citations
  4. SPMDM: Enhancing Masked Diffusion Models through Simplifying Sampling PathYichen Zhu, Weiyu Chen, James Kwok, Zhou ZhaoNeurIPS 2025 · 1 citation
  5. Diffusion with Truncated Blocks: Fast and High-Quality Text Generation using Truncated Block GenerationYuyan Zhou, Weiyu Chen, James T. KwokACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines