Particles Don’t Care About Z: Towards Scaling Entropy Estimation of Unnormalized Densities

Computing the differential entropy of distributions known only up to a normalization constant is a fundamental challenge with broad theoretical and practical significance. While variational inference is highly scalable for density approximation from samples, its application to unnormalized densities remains under-explored due to the difficulty of constructing variational distributions that exploit the structure of the unnormalized density and are simultaneously expressive, tractable, efficiently samplable. Recently, Messaoud et al. (ICLR 24) introduced P-SVGD, a Stein variational inference method for this setting. We show, however, that P-SVGD fails to scale to high dimensions due to incorrect invertibility assumptions, omission of a critical trace-of-Hessian term, and unstable divergence-control heuristics. We propose MET-SVGD, a principled extension of P-SVGD that provides a general framework for stable SVGD hyperparameter selection with global invertibility and convergence guarantees. Empirically, MET-SVGD achieves up to $12\times$ and $16\times$ lower entropy estimation error than P-SVGD and existing SVGD baselines, respectively. On CIFAR-10 energy-based image generation, it improves FID by $80.4\%$ and yields $64\times$ higher training stability. In maximum-entropy reinforcement learning, it achieves up to $16\%$ higher returns than P-SVGD. Code is available at https://shorturl.at/fTG0G.