EMNLP Findings2023
InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective
Yifan Song, Peiyi Wang, Weimin Xiong, Dawei Zhu, Tianyu Liu, Zhifang Sui, Sujian Li
Abstract
Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks. We focus on continual text classification under the class-incremental setting. Recent CL studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. In this paper, through an in-depth exploration of the representation learning process in CL, we discover that the compression effect of the information bottleneck leads to confusion on analogous classes. To enable the model learn more sufficient representations, we propose a novel replay-based continual text classification method, InfoCL. Our approach utilizes fast-slow and currentpast contrastive learning to perform mutual information maximization and better recover the previously learned representations. In addition, InfoCL incorporates an adversarial memory augmentation strategy to alleviate the overfitting problem of replay. Experimental results demonstrate that InfoCL effectively mitigates forgetting and achieves state-of-the-art performance on three text classification tasks. The code is publicly available at https://github. com/Yifan-Song793/InfoCL.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Co2L: Contrastive Continual Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Co2L: Contrastive Continual Learning cites Supervised Contrastive Learning
- Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Mitigating Non-Representative Prototypes and Representation Bias in Few-Shot Continual Relation Extraction cites Supervised Contrastive Learning
Cited by top-tier papers2
- ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled TuningJinyang Zhang, Yue Fang, Hongxin Ding, Weibin Liao et al.ICLR 2026 · 5 citations
- Mitigating Non-Representative Prototypes and Representation Bias in Few-Shot Continual Relation ExtractionThanh Duc Pham, Nam Le Hai, Linh Ngo Van, Nguyen Thi Ngoc Diep et al.ACL 2025
Builds on20
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- FreeLB: Enhanced Adversarial Training for Natural Language UnderstandingChen Zhu, Yu Cheng, Zhe Gan, Siqi Sun et al.ICLR 2020 · 502 citations
- Co2L: Contrastive Continual LearningHyuntak Cha, Jaeho Lee, Jinwoo ShinICCV 2021 · 391 citations
- LAMOL: LAnguage MOdeling for Lifelong Language LearningFan-Keng Sun, Cheng-Hao Ho, Hung-Yi LeeICLR 2020 · 247 citations
Related papers
- PCR: Proxy-Based Contrastive Replay for Online Class-Incremental Continual LearningHuiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li et al.CVPR 2023
- Dealing with Cross-Task Class Discrimination in Online Continual LearningYiduo Guo, Bing Liu, Dongyan ZhaoCVPR 2023
- Prototype-Sample Relation Distillation: Towards Replay-Free Continual LearningNader Asadi, MohammadReza Davari, Sudhir P. Mudur, Rahaf Aljundi et al.ICML 2023 · 61 citations
- Defying Imbalanced Forgetting in Class Incremental LearningShixiong Xu, Gaofeng Meng, Xing Nie, Bolin Ni et al.AAAI 2024 · 8 citations
- AttriCLIP: A Non-Incremental Learner for Incremental Knowledge LearningRunqi Wang, Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu et al.CVPR 2023
