Lune

NeurIPS2024Top-tier venue

Simple and Effective Masked Diffusion Language Models

Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander M. Rush, Volodymyr Kuleshov

Abstract

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

303 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models cites Large Language Diffusion Models
  • LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models cites Simplified and Generalized Masked Diffusion for Discrete Data
  • LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Structured Denoising Diffusion Models in Discrete State-Spaces cites Denoising Diffusion Probabilistic Models
  • Large Language Diffusion Models cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • Large Language Diffusion Models cites Diffusion-LM Improves Controllable Text Generation
  • Large Language Diffusion Models cites Scalable Diffusion Models with Transformers
  • Large Language Diffusion Models cites Simplified and Generalized Masked Diffusion for Discrete Data
  • Large Language Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Large Language Diffusion Models
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Diffusion-LM Improves Controllable Text Generation
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Scalable Diffusion Models with Transformers
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Simplified and Generalized Masked Diffusion for Discrete Data
  • Remasking Discrete Diffusion Models with Inference-Time Scaling cites Denoising Diffusion Probabilistic Models
  • Diffusion-LM Improves Controllable Text Generation cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • Diffusion-LM Improves Controllable Text Generation cites Denoising Diffusion Probabilistic Models
  • Scalable Diffusion Models with Transformers cites Denoising Diffusion Probabilistic Models
  • Simplified and Generalized Masked Diffusion for Discrete Data cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • Simplified and Generalized Masked Diffusion for Discrete Data cites Diffusion-LM Improves Controllable Text Generation
  • Simplified and Generalized Masked Diffusion for Discrete Data cites Scalable Diffusion Models with Transformers
  • Simplified and Generalized Masked Diffusion for Discrete Data cites Denoising Diffusion Probabilistic Models
  • Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding cites LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
  • Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding cites Structured Denoising Diffusion Models in Discrete State-Spaces
  • Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding cites Large Language Diffusion Models
  • Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding cites Simplified and Generalized Masked Diffusion for Discrete Data
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers303

  1. Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
  2. Simplified and Generalized Masked Diffusion for Discrete DataJiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet et al.NeurIPS 2024 · 693 citations
  3. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel DecodingChengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu et al.ICLR 2026 · 428 citations
  4. LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion ModelsFengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang et al.ACL 2026 · 229 citations
  5. Remasking Discrete Diffusion Models with Inference-Time ScalingGuanghan Wang, Yair Schiff, Subham S. Sahoo, Volodymyr KuleshovNeurIPS 2025 · 199 citations

Builds on36

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
  3. Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
  4. Structured Denoising Diffusion Models in Discrete State-SpacesJacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow et al.NeurIPS 2021 · 2,256 citations
  5. Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations

Related papers

  1. Dual-objective Language Models: Training Efficiency Without OverfittingDavid Samuel, Lucas Georges Gabriel CharpentierICLR 2026
  2. Scaling Beyond Masked Diffusion Language ModelsSubham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux et al.ICML 2026 · 18 citations
  3. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language ModelsMarianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang et al.ICLR 2025
  4. Unifying Masked Diffusion Models with Various Generation Orders and BeyondChunsan Hong, Sanghyun Lee, Jong Chul YEICML 2026
  5. A Cheaper and Better Diffusion Language Model with Soft-Masked NoiseJiaao Chen, Aston Zhang, Mu Li, Alex Smola et al.EMNLP 2023 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines