Lune

ICML2025Top-tier venue

Do Bayesian Neural Networks Actually Behave Like Bayesian Models?

Gábor Pituk, Vik Shirvaikar, Tom Rainforth

Abstract

We empirically investigate how well popular approximate inference algorithms for Bayesian Neural Networks (BNNs) respect the theoretical properties of Bayesian belief updating. We find strong evidence on synthetic regression and real-world image classification tasks that common BNN algorithms such as variational inference, Laplace approximation, SWAG, and SGLD fail to update in a consistent manner, forget about old data under sequential updates, and violate the predictive coherence properties that would be expected of Bayesian methods. These observed behaviors imply that care should be taken when treating BNNs as true Bayesian models, particularly when using them beyond static prediction settings, such as for active, continual, or transfer learning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • What Are Bayesian Neural Network Posteriors Really Like? cites Bayesian Deep Learning and a Probabilistic Perspective of Generalization
  • Bayesian Deep Learning and a Probabilistic Perspective of Generalization cites What Are Bayesian Neural Network Posteriors Really Like?
  • What Are Bayesian Neural Network Posteriors Really Like? cites Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
  • Laplace Redux - Effortless Bayesian Deep Learning cites What Are Bayesian Neural Network Posteriors Really Like?
  • Laplace Redux - Effortless Bayesian Deep Learning cites Bayesian Deep Learning and a Probabilistic Perspective of Generalization
  • Laplace Redux - Effortless Bayesian Deep Learning cites Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
  • How Good is the Bayes Posterior in Deep Neural Networks Really? cites Bayesian Deep Learning and a Probabilistic Perspective of Generalization
  • How Good is the Bayes Posterior in Deep Neural Networks Really? cites Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on20

  1. Bayesian Deep Learning and a Probabilistic Perspective of GeneralizationAndrew Gordon Wilson, Pavel IzmailovNeurIPS 2020 · 845 citations
  2. Laplace Redux - Effortless Bayesian Deep LearningErik A. Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen et al.NeurIPS 2021 · 508 citations
  3. What Are Bayesian Neural Network Posteriors Really Like?Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, Andrew Gordon WilsonICML 2021 · 458 citations
  4. How Good is the Bayes Posterior in Deep Neural Networks Really?Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiatkowski et al.ICML 2020 · 409 citations
  5. Cyclical Stochastic Gradient MCMC for Bayesian Deep LearningRuqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen et al.ICLR 2020 · 292 citations

Related papers

  1. Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution ShiftFlorian Seligmann, Philipp Becker, Michael Volpp, Gerhard NeumannNeurIPS 2023 · 32 citations
  2. On the Expressiveness of Approximate Inference in Bayesian Neural NetworksAndrew Y. K. Foong, David R. Burt, Yingzhen Li, Richard E. TurnerNeurIPS 2020 · 142 citations
  3. Reparameterization invariance in approximate Bayesian inferenceHrittik Roy, Marco Miani, Carl Henrik Ek, Philipp Hennig et al.NeurIPS 2024 · 20 citations
  4. Continual Learning with Bayesian Neural Networks for Non-Stationary DataRichard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt et al.ICLR 2020 · 82 citations
  5. Addressing Catastrophic Forgetting in Few-Shot ProblemsPau Ching Yap, Hippolyt Ritter, David BarberICML 2021 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines