ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
Sagnik Bhattacharya, Abhiram Rao Gorle, Ahsan Bilal, Connor Ding, Amit Kumar Singh Yadav, Tsachy Weissman
Abstract
Generative modeling of non-negative, discrete data, such as symbolic music, remains challenging due to two persistent limitations in existing methods. First, most approaches rely on modeling continuous embeddings, which are not wellsuited for inherently discrete data distributions. Second, they typically optimize variational lower bounds instead of the true data likelihood, leading to inaccurate likelihood estimates and degraded sampling quality. While recent diffusion-based models have addressed these issues individually, we tackle them jointly. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), inspired by photon arrival processes, unifying exact likelihood estimation with discrete-state generative modeling. Central to our approach is an information-theoretic Poisson Reconstruction Loss (PRL) that admits a provable, exact relationship with the true data likelihood. ItDPDM achieves improved likelihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, ItDPDM attains superior likelihood estimates and competitive generation quality, demonstrating a proof of concept for principled, distribution-robust discrete generative modeling.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- Improved Denoising Diffusion Probabilistic Models cites Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis cites Improved Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Improved Denoising Diffusion Probabilistic Models
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Diffusion Models Beat GANs on Image Synthesis
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Denoising Diffusion Probabilistic Models
- Learning normalized image densities via dual score matching cites Improved Denoising Diffusion Probabilistic Models
- Learning normalized image densities via dual score matching cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on18
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
- Improved Denoising Diffusion Probabilistic ModelsAlexander Quinn Nichol, Prafulla DhariwalICML 2021 · 5,234 citations
Related papers
- Score-based Continuous-time Discrete Diffusion ModelsHaoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans et al.ICLR 2023 · 7 citations
- Diffusing Gaussian Mixtures for Generating Categorical DataFlorence Regol, Mark CoatesAAAI 2023 · 6 citations
- Information-Theoretic DiffusionXianghao Kong, Rob Brekelmans, Greg Ver SteegICLR 2023
- Discrete Contrastive Diffusion for Cross-Modal Music and Image GenerationYe Zhu, Yu Wu, Kyle Olszewski, Jian Ren et al.ICLR 2023 · 10 citations
- Structured Denoising Diffusion Models in Discrete State-SpacesJacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow et al.NeurIPS 2021 · 2,256 citations
