Lune

ACL Findings2026

Diffusion with Truncated Blocks: Fast and High-Quality Text Generation using Truncated Block Generation

Yuyan Zhou, Weiyu Chen, James T. Kwok

Abstract

Diffusion-based Large Language Models (dLLMs) are emerging as a powerful alternative to traditional autoregressive models. These models learn to generate text by iteratively denoising masked sequences. In this work, we identify a critical problem in dLLMs: the model's attention is wastefully expended on uninformative mask tokens, diluting its focus on meaningful context. We term this phenomenon "attention dilution". We further show that this artifact is amplified by token-level noising, whereas models employing sequence-level noise exhibit a reduced effect. To resolve this problem, we introduce Truncated Block Generation, a novel sampling algorithm that not only mitigates attention dilution but also enables faster inference and flexible-length sequence generation. Extensive experiments validate our analysis and demonstrate the marked effectiveness of our proposed method in enhancing both the performance and efficiency of dLLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Diffusion Models cites Diffusion-LM Improves Controllable Text Generation
  • Large Language Diffusion Models cites Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
  • Large Language Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Diffusion-LM Improves Controllable Text Generation cites Denoising Diffusion Probabilistic Models
  • Elucidating the Design Space of Diffusion-Based Generative Models cites Denoising Diffusion Probabilistic Models
  • Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution cites Diffusion-LM Improves Controllable Text Generation
  • Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Elucidating the Design Space of Diffusion-Based Generative ModelsTero Karras, Miika Aittala, Timo Aila, Samuli LaineNeurIPS 2022 · 3,959 citations
  3. Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations
  4. Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
  5. Discrete Diffusion Modeling by Estimating the Ratios of the Data DistributionAaron Lou, Chenlin Meng, Stefano ErmonICML 2024 · 473 citations

Related papers

  1. Learning Unmasking Policies for Diffusion Language ModelsMetod Jazbec, Theo X. Olausson, Louis Béthune, Pierre Ablin et al.ICML 2026 · 24 citations
  2. Attention Sinks in Diffusion Language ModelsMaximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri et al.ACL Findings 2026
  3. Theoretical Benefit and Limitation of Diffusion Language ModelGuhao Feng, Yihan Geng, Jian Guan, Wei Wu et al.NeurIPS 2025 · 52 citations
  4. Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language ModelsJinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao et al.ICLR 2026 · 34 citations
  5. Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in SpeedYonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong et al.ICML 2026 · 22 citations
Dusk over the sea between two cliffs drawn in fine vertical lines