ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model

Generative modeling of non-negative, discrete data, such as symbolic music, remains challenging due to two persistent limitations in existing methods. First, most approaches rely on modeling continuous embeddings, which are not wellsuited for inherently discrete data distributions. Second, they typically optimize variational lower bounds instead of the true data likelihood, leading to inaccurate likelihood estimates and degraded sampling quality. While recent diffusion-based models have addressed these issues individually, we tackle them jointly. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), inspired by photon arrival processes, unifying exact likelihood estimation with discrete-state generative modeling. Central to our approach is an information-theoretic Poisson Reconstruction Loss (PRL) that admits a provable, exact relationship with the true data likelihood. ItDPDM achieves improved likelihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, ItDPDM attains superior likelihood estimates and competitive generation quality, demonstrating a proof of concept for principled, distribution-robust discrete generative modeling.