What Is It Like to Be a Noise? An Entropy-based Gaussian Noise Regularization for Diffusion Models

Inference-time optimization of diffusion latents enables powerful control but often degrades the statistical structure of true Gaussian noise, causing artifacts and reward hacking. To address this, we propose a Gaussianity regularizer that aligns a sample's local statistics with a typical Gaussian realization, rather than relying on pointwise likelihood. We formalize this by computing the KL divergence between the sample distribution and the Gaussian prior. To make the divergence computation tractable from a single sample, we lift each candidate latent into an empirical distribution induced by its statistics and model it as a pairwise Markov Random Field. This yields a Bethe-Kikuchi-style regularizer with 1D marginal, 2D spatial, and multi-scale terms. Our results show improved latent optimization stability and generation quality over prior approaches.