Overcoming Catastrophic Forgetting by Bayesian Generative Regularization
Pei-Hung Chen, Wei Wei, Cho-Jui Hsieh, Bo Dai
摘要
The streaming update of Bayesian posterior calculation provides us a natural way for continual learning. However, the naïve mean-field posterior parametrization for variational approximation is inappropiate in neural network, and thus, lock the full ability for preventing catastrophic forgetting. To resolve this issue, we introduce a generative regularization for all given classification models, which is implemented by leveraging energy-based models with contrastive loss, to obtain the sufficient features for valid decomposition in posterior approxiamtion. By combining discriminative and generative loss together, we empirically show that the proposed method outperforms state-ofthe-art methods on a variety of tasks, avoiding catastrophic forgetting in continual learning. In particular, the proposed method outperforms baseline methods over 15% on the Fashion-MNIST dataset and 10% on the CUB dataset.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
引用它的顶会 Paper1
它引用的 Paper2
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 次引用
- Your classifier is secretly an energy based model and you should treat it like oneWill Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud 等ICLR 2020 · 643 次引用
相关 Paper
- BooVAE: Boosting Approach for Continual Learning of VAEEvgenii Egorov, Anna Kuzina, Evgeny BurnaevNeurIPS 2021 · 34 次引用
- Bayesian Structural Adaptation for Continual LearningAbhishek Kumar, Sunabha Chatterjee, Piyush RaiICML 2021 · 7 次引用
- Uncertainty-guided Continual Learning with Bayesian Neural NetworksSayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, Marcus RohrbachICLR 2020 · 211 次引用
- Temporal-Difference Variational Continual LearningLuckeciano Carvalho Melo, Alessandro Abate, Yarin GalNeurIPS 2025 · 1 次引用
- Variational Auto-Regressive Gaussian Processes for Continual LearningSanyam Kapoor, Theofanis Karaletsos, Thang D. BuiICML 2021 · 32 次引用
