ACL Findings2026
DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models
Abstract
Masked diffusion language models (MDLMs) have recently emerged as a new paradigm in language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation. Specifically, DOS exploits attention matrices from transformer blocks to approximate inter-token dependencies, emphasizing information from unmasked tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Improved Denoising Diffusion Probabilistic Models cites Language Models are Few-Shot Learners
- Improved Denoising Diffusion Probabilistic Models cites Denoising Diffusion Probabilistic Models
- Improved Denoising Diffusion Probabilistic Models cites Score-Based Generative Modeling through Stochastic Differential Equations
- Score-Based Generative Modeling through Stochastic Differential Equations cites Denoising Diffusion Probabilistic Models
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Improved Denoising Diffusion Probabilistic ModelsAlexander Quinn Nichol, Prafulla DhariwalICML 2021 · 5,234 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Score-Based Generative Modeling through Stochastic Differential EquationsYang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar et al.ICLR 2021 · 1,270 citations
Related papers
- Improving Sampling for Masked Diffusion Models via Information GainKaisen Yang, Jayden Teoh, Kaicheng Yang, Yitong Zhang et al.ICML 2026 · 6 citations
- DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMsBumjun Kim, Dongjae Jeon, Moongyu Jeon, Albert NoICML 2026 · 8 citations
- Plan for Speed: Dilated Scheduling for Masked Diffusion Language ModelsOmer Luxembourg, Haim Permuter, Eliya NachmaniICML 2026 · 30 citations
- Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language ModelsJia Deng, Junyi Li, Xin Zhao, Jinpeng Wang et al.ACL 2026
- DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial AttentionYounjoo Lee, Seungkyun Dan, Junghoo Lee, Jaiyoung Park et al.ICML 2026 · 2 citations
