ACL Findings2026
Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion
Tarun Kathuria, Sachin Kumar
Abstract
We present a discrete diffusion-based language model using Glauber dynamics from statistical physics. Our main insight is that instead of trying to train a discrete state space diffusion model using Glauber dynamics with a uniform transition kernel as the forward process, one can set up an "energy function" based on pretrained causal/masked language models. When viewed as the stationary distribution, this energy function allows us to significantly improve the quality of the generated text. Incorporating UL2 as the pretrained model into our diffusion pipeline, we outperform prior diffusion based LMs and perform competitively with autoregressive models of comparable model sizes. Furthermore, our models are competitive with or outperform prior diffusion models and GPT-2 style auto-regressive models on zeroshot common sense reasoning tasks as well as planning and search tasks like Sudoku and Zebra puzzles.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Improved Denoising Diffusion Probabilistic Models cites Denoising Diffusion Probabilistic Models
- Video Diffusion Models cites Improved Denoising Diffusion Probabilistic Models
- Video Diffusion Models cites Denoising Diffusion Probabilistic Models
- Scalable Diffusion Models with Transformers cites Improved Denoising Diffusion Probabilistic Models
- Scalable Diffusion Models with Transformers cites Denoising Diffusion Probabilistic Models
Builds on27
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
- Improved Denoising Diffusion Probabilistic ModelsAlexander Quinn Nichol, Prafulla DhariwalICML 2021 · 5,234 citations
- Video Diffusion ModelsJonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan et al.NeurIPS 2022 · 2,948 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
Related papers
- Glauber Generative Model: Discrete Diffusion Models via Binary ClassificationHarshit Varma, Dheeraj Mysore Nagaraj, Karthikeyan ShanmugamICLR 2025
- DiffusionBERT: Improving Generative Masked Language Models with Diffusion ModelsZhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang et al.ACL 2023 · 63 citations
- A Cheaper and Better Diffusion Language Model with Soft-Masked NoiseJiaao Chen, Aston Zhang, Mu Li, Alex Smola et al.EMNLP 2023 · 16 citations
- Scaling Diffusion Language Models via Adaptation from Autoregressive ModelsShansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye et al.ICLR 2025
- Beyond Autoregression: Discrete Diffusion for Complex Reasoning and PlanningJiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng et al.ICLR 2025
