Lune

ACL Findings2026

One Token Is Enough: Improving Diffusion Language Models with a Sink Token

Zihou Zhang, Zheyong Xie, Li Zhong, Haifeng Liu, Shaosheng Cao

Abstract

Diffusion Language Models (DLMs) have emerged as a compelling alternative to autoregressive approaches, enabling parallel text generation with competitive performance. Despite these advantages, there is a critical instability in DLMs: the moving sink phenomenon. Our analysis indicates that sink tokens exhibit low-norm representations in the Transformer's value space, and that the moving sink phenomenon serves as a protective mechanism in DLMs to prevent excessive information mixing. However, their unpredictable positions across diffusion steps undermine inference robustness. To resolve this, we propose a simple but effective extra sink token implemented via a modified attention mask. Specifically, we introduce a special token constrained to attend solely to itself, while remaining globally visible to all other tokens. Experimental results demonstrate that introducing a single extra token stabilizes attention sinks, substantially improving model performance. Crucially, further analysis confirms that the effectiveness of this token is independent of its position and characterized by negligible semantic content, validating its role as a robust and dedicated structural sink. Our code is available at https://github.com/ skywalker0523/OneTokenIsEnough.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Streaming Language Models with Attention Sinks cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Large Language Diffusion Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  4. Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
  5. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMsSuyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang et al.ICLR 2024 · 432 citations

Related papers

  1. Attention Sinks in Diffusion Language ModelsMaximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri et al.ACL Findings 2026
  2. When Attention Sink Emerges in Language Models: An Empirical ViewXiangming Gu, Tianyu Pang, Chao Du, Qian Liu et al.ICLR 2025
  3. Attention Sinks in Diffusion Transformers: A Causal AnalysisFANGZHENG WU, Brian SummaICML 2026
  4. The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension DisparitySiquan Li, Kaiqi Jiang, Jiacheng Sun, Tianyang HuICML 2026 · 1 citation
  5. A Single Layer to Explain Them All: Understanding Massive Values in Large Language ModelsZeru Shi, Zhenting Wang, Fan Yang, Qifan Wang et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines