ACL Findings2026
Attention Sinks in Diffusion Language Models
Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, Alessio Devoto
Abstract
Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. Although their efficiency and effectiveness have been extensively studied, the internal mechanisms that govern DLMs remain largely unexplored. In this work, we conduct an empirical analysis of DLM attention patterns, focusing on the attention sinking phenomenon, an effect previously observed in various transformer-based architectures. Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. First, unlike in ARMs, the sink positions in DLMs tend to shift throughout the generation process, displaying a dynamic behaviour. Second, while ARMs are highly sensitive to the removal of attention sinks, DLMs remain robust: masking sinks leads to only a minor degradation in performance. These results provide new insights into the inner workings of diffusion-based language models and highlight fundamental differences in how they allocate and utilize attention compared to autoregressive models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Attention Sinks in Diffusion Transformers: A Causal Analysis cites Efficient Streaming Language Models with Attention Sinks
- Large Language Diffusion Models cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Large Language Diffusion Models cites Diffusion-LM Improves Controllable Text Generation
- Large Language Diffusion Models cites Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
- Diffusion-LM Improves Controllable Text Generation cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink cites Efficient Streaming Language Models with Attention Sinks
- Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing cites Efficient Streaming Language Models with Attention Sinks
- Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing cites Large Language Diffusion Models
- Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing cites Diffusion-LM Improves Controllable Text Generation
Cited by top-tier papers3
- Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention SinkGuozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo et al.ICML 2026 · 5 citations
- Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context FocusingLingkun Long, Yushi Huang, Shihao Bai, Ruihao Gong et al.ACL 2026 · 3 citations
- Attention Sinks in Diffusion Transformers: A Causal AnalysisFANGZHENG WU, Brian SummaICML 2026
Builds on27
- Structured Denoising Diffusion Models in Discrete State-SpacesJacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow et al.NeurIPS 2021 · 2,256 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations
- Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
- Argmax Flows and Multinomial Diffusion: Learning Categorical DistributionsEmiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré et al.NeurIPS 2021 · 782 citations
Related papers
- One Token Is Enough: Improving Diffusion Language Models with a Sink TokenZihou Zhang, Zheyong Xie, Li Zhong, Haifeng Liu et al.ACL Findings 2026
- Revealing the Attention Floating Mechanism in Masked Diffusion ModelsXin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang et al.ACL Findings 2026
- Masks Can Be Distracting: On Context Comprehension in Diffusion Language ModelsJulianna Piskorz, Cristina Pinneri, Alvaro Correia, Motasem Alfarra et al.ICML 2026 · 5 citations
- Diffusion with Truncated Blocks: Fast and High-Quality Text Generation using Truncated Block GenerationYuyan Zhou, Weiyu Chen, James T. KwokACL Findings 2026
- Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in SpeedYonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong et al.ICML 2026 · 22 citations
