Lune

ICML2025Top-tier venue

Understanding the difficulties of posterior predictive estimation

Abhinav Agrawal, Justin Domke

Abstract

Predictive posterior densities (PPDs) are essential in approximate inference for quantifying predictive uncertainty and comparing inference methods. Typically, PPDs are estimated by simple Monte Carlo (MC) averages. In this paper, we expose a critical under-recognized issue: the signal-tonoise ratio (SNR) of the simple MC estimator can sometimes be extremely low, leading to unreliable estimates. Our main contribution is a theoretical analysis demonstrating that even with exact inference, SNR can decay rapidly with an increase in (a) the mismatch between training and test data, (b) the dimensionality of the latent space, or (c) the size of test data relative to training data.

Through several examples, we empirically verify these claims and show that these factors indeed lead to poor SNR and unreliable PPD estimates (sometimes, estimates are off by hundreds of nats even with a million samples). While not the primary focus, we also explore an adaptive importance sampling approach as an illustrative way to mitigate the problem, where we learn the proposal distribution by maximizing a variational proxy to the SNR. Taken together, our findings highlight an important challenge and provide essential insights for reliable estimation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Bayesian Model Selection, the Marginal Likelihood, and Generalization cites What Are Bayesian Neural Network Posteriors Really Like?
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on20

  1. Decision-Making with Auto-Encoding Variational BayesRomain Lopez, Pierre Boyeau, Nir Yosef, Michael I. Jordan et al.NeurIPS 2020 · 22,845 citations
  2. What Are Bayesian Neural Network Posteriors Really Like?Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, Andrew Gordon WilsonICML 2021 · 458 citations
  3. Bayesian Model Selection, the Marginal Likelihood, and GeneralizationSanae Lotfi, Pavel Izmailov, Gregory W. Benton, Micah Goldblum et al.ICML 2022 · 83 citations
  4. Improved off-policy training of diffusion samplersMarcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos et al.NeurIPS 2024 · 52 citations
  5. Dangers of Bayesian Model Averaging under Covariate ShiftPavel Izmailov, Patrick Nicholson, Sanae Lotfi, Andrew Gordon WilsonNeurIPS 2021 · 51 citations

Related papers

  1. On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian ProcessesTim G. J. Rudner, Oscar Key, Yarin Gal, Tom RainforthICML 2021 · 4 citations
  2. On the Expressiveness of Approximate Inference in Bayesian Neural NetworksAndrew Y. K. Foong, David R. Burt, Yingzhen Li, Richard E. TurnerNeurIPS 2020 · 142 citations
  3. On the difficulty of unbiased alpha divergence minimizationTomas Geffner, Justin DomkeICML 2021 · 20 citations
  4. Parametric Gaussian Process RegressorsMartin Jankowiak, Geoff Pleiss, Jacob R. GardnerICML 2020 · 82 citations
  5. Posterior Refinement Improves Sample Efficiency in Bayesian Neural NetworksAgustinus Kristiadi, Runa Eschenhagen, Philipp HennigNeurIPS 2022 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines