EMNLP Findings2023
InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation
Renzhi Wang, Jing Li, Piji Li
Abstract
Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose In-foDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfofirst" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Structured Denoising Diffusion Models in Discrete State-Spaces cites Improved Denoising Diffusion Probabilistic Models
- Structured Denoising Diffusion Models in Discrete State-Spaces cites Denoising Diffusion Probabilistic Models
- Improved Denoising Diffusion Probabilistic Models cites Denoising Diffusion Probabilistic Models
- Diffusion-LM Improves Controllable Text Generation cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Diffusion-LM Improves Controllable Text Generation cites Improved Denoising Diffusion Probabilistic Models
- Diffusion-LM Improves Controllable Text Generation cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on13
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Improved Denoising Diffusion Probabilistic ModelsAlexander Quinn Nichol, Prafulla DhariwalICML 2021 · 5,234 citations
- Structured Denoising Diffusion Models in Discrete State-SpacesJacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow et al.NeurIPS 2021 · 2,256 citations
- Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations
Related papers
- Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion ProcessesBocheng Li, Zhujin Gao, Linli XuACL 2025
- Text Diffusion with Reinforced ConditioningYuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang et al.AAAI 2024 · 2 citations
- AR-Diffusion: Auto-Regressive Diffusion Model for Text GenerationTong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng et al.NeurIPS 2023 · 170 citations
- DiffuSeq: Sequence to Sequence Text Generation with Diffusion ModelsShansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu et al.ICLR 2023 · 94 citations
- TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model EncodingsAlexander Shabalin, Viacheslav Meshchaninov, Egor Chimbulatov, Vladislav Lapikov et al.AAAI 2025 · 11 citations
