ACL Findings2023
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
Genta Indra Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro
Abstract
Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models cites Unsupervised Cross-lingual Representation Learning at Scale
- MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages cites Unsupervised Cross-lingual Representation Learning at Scale
Cited by top-tier papers7
- Demystifying Language Model Forgetting with Low-rank Example AssociationsXisen Jin, Xiang RenNeurIPS 2025 · 9 citations
- Rethinking Optimization and Architecture for Tiny Language ModelsYehui Tang, Kai Han, Fangcheng Liu, Yunsheng Ni et al.ICML 2024 · 8 citations
- Breaking the Curse of Multilinguality with Cross-lingual Expert Language ModelsTerra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li et al.EMNLP 2024 · 6 citations
- Breaking Language Barriers: Cross-Lingual Continual Pre-Training at ScaleWenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin et al.EMNLP 2024 · 3 citations
- Bridging the Language Gap: Uncovering and Aligning Shared Circuits for Multi-Hop Reasoning in Multilingual LLMsChenghao Sun, Zhen Huang, Yonggang Zhang, Xinmei Tian et al.AAAI 2026
Builds on4
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse LanguagesJack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie et al.ACL 2023 · 88 citations
- Continual Learning in Task-Oriented Dialogue SystemsAndrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon et al.EMNLP 2021 · 68 citations
- Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot LearningXisen Jin, Bill Yuchen Lin, Mohammad Rostami, Xiang RenEMNLP Findings 2021 · 29 citations
Related papers
- Orthogonal Subspace Learning for Language Model Continual LearningXiao Wang, Tianze Chen, Qiming Ge, Han Xia et al.EMNLP Findings 2023 · 79 citations
- Compositional Language Continual LearningYuanpeng Li, Liang Zhao, Kenneth Church, Mohamed ElhoseinyICLR 2020 · 40 citations
- Parameter-Efficient Finetuning for Robust Continual Multilingual LearningKartikeya Badola, Shachi Dave, Partha P. TalukdarACL Findings 2023 · 6 citations
- Progressive Prompts: Continual Learning for Language ModelsAnastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa et al.ICLR 2023 · 15 citations
- Multimodal Continual Instruction Tuning with Dynamic Gradient GuidanceSongze Li, Mingyu Gao, Tonghua Su, Xu-Yao Zhang et al.CVPR 2026 · 6 citations
