Lune

ICML2021Top-tier venue

Overcoming Catastrophic Forgetting by Bayesian Generative Regularization

Pei-Hung Chen, Wei Wei, Cho-Jui Hsieh, Bo Dai

Abstract

The streaming update of Bayesian posterior calculation provides us a natural way for continual learning. However, the naïve mean-field posterior parametrization for variational approximation is inappropiate in neural network, and thus, lock the full ability for preventing catastrophic forgetting. To resolve this issue, we introduce a generative regularization for all given classification models, which is implemented by leveraging energy-based models with contrastive loss, to obtain the sufficient features for valid decomposition in posterior approxiamtion. By combining discriminative and generative loss together, we empirically show that the proposed method outperforms state-ofthe-art methods on a variety of tasks, avoiding catastrophic forgetting in continual learning. In particular, the proposed method outperforms baseline methods over 15% on the Fashion-MNIST dataset and 10% on the CUB dataset.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Do Bayesian Neural Networks Actually Behave Like Bayesian Models?Gábor Pituk, Vik Shirvaikar, Tom RainforthICML 2025

Builds on2

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Your classifier is secretly an energy based model and you should treat it like oneWill Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud et al.ICLR 2020 · 643 citations

Related papers

  1. BooVAE: Boosting Approach for Continual Learning of VAEEvgenii Egorov, Anna Kuzina, Evgeny BurnaevNeurIPS 2021 · 34 citations
  2. Bayesian Structural Adaptation for Continual LearningAbhishek Kumar, Sunabha Chatterjee, Piyush RaiICML 2021 · 7 citations
  3. Uncertainty-guided Continual Learning with Bayesian Neural NetworksSayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, Marcus RohrbachICLR 2020 · 211 citations
  4. Temporal-Difference Variational Continual LearningLuckeciano Carvalho Melo, Alessandro Abate, Yarin GalNeurIPS 2025 · 1 citation
  5. Variational Auto-Regressive Gaussian Processes for Continual LearningSanyam Kapoor, Theofanis Karaletsos, Thang D. BuiICML 2021 · 32 citations
Dusk over the sea between two cliffs drawn in fine vertical lines