Lune

NeurIPS2024Top-tier venue

Learn more, but bother less: parameter efficient continual learning

Fuli Qiao, Mehrdad Mahdavi

Abstract

Large Language Models (LLMs) have demonstrated profound capabilities due to their extensive pre-training on diverse corpora. However, LLMs often struggle with catastrophic forgetting when engaged in sequential task learning. In this paper, we propose a novel parameter-efficient approach for continual learning in LLMs, which empirically investigates knowledge transfer from previously learned tasks to new tasks through low-rank matrix parameters, enhancing the learning of new tasks without significant interference. Our method employs sensitivity-based analysis of low-rank matrix parameters to identify knowledge-specific parameters between sequential tasks, which are used to initialize the low-rank matrix parameters in new tasks. To maintain orthogonality and minimize forgetting, we further involve the gradient projection technique that keeps the low-rank subspaces of each new task orthogonal to those of previous tasks. Our experimental results on continual learning benchmarks validate the efficacy of our proposed method, which outperforms existing state-of-the-art methods in reducing forgetting, enhancing task performance, and preserving the model's ability to generalize to unseen tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

10 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Revisiting Weight Regularization for Low-Rank Continual Learning cites Learning to Prompt for Continual Learning
  • Any-SSR: How Recursive Least Squares Works in Continual Learning of Large Language Models cites Learning to Prompt for Continual Learning
  • Any-SSR: How Recursive Least Squares Works in Continual Learning of Large Language Models cites LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
  • Learning to Prompt for Continual Learning cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • Towards a Unified View of Parameter-Efficient Transfer Learning cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • Little By Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts cites Learning to Prompt for Continual Learning
  • Little By Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts cites LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
  • Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image Analysis cites Learning to Prompt for Continual Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers10

  1. Revisiting Weight Regularization for Low-Rank Continual LearningYaoyue Zheng, Yin Zhang, Joost van de Weijer, Gido van de Ven et al.ICLR 2026 · 7 citations
  2. Any-SSR: How Recursive Least Squares Works in Continual Learning of Large Language ModelsKai Tong, Kang Pan, Xiao Zhang, Erli Meng et al.ICCV 2025 · 3 citations
  3. Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image AnalysisByung Hyun Lee, Wongi Jeong, Woojae Han, Kyoungbun Lee et al.ICCV 2025 · 3 citations
  4. Little By Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory ExpertsHaodong Lu, Chongyang Zhao, Minhui Xue, Lina Yao et al.ICML 2026 · 2 citations
  5. Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual LearningZiqi Jia, Anmin Wang, Xiaoyang Qu, Xiaowen Yang et al.ACL 2025

Builds on19

  1. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  2. Learning to Prompt for Continual LearningZifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang et al.CVPR 2022 · 635 citations
  3. LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5Chengwei Qin, Shafiq R. JotyICLR 2022 · 128 citations
  4. TRGP: Trust Region Gradient Projection for Continual LearningSen Lin, Li Yang, Deliang Fan, Junshan ZhangICLR 2022 · 107 citations
  5. The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations

Related papers

  1. Orthogonal Subspace Learning for Language Model Continual LearningXiao Wang, Tianze Chen, Qiming Ge, Han Xia et al.EMNLP Findings 2023 · 79 citations
  2. Merge before Forget: A Single LoRA Continual Learning via Continual MergingFuli Qiao, Mehrdad MahdaviICLR 2026 · 11 citations
  3. Continual Gradient Low-Rank Projection Fine-Tuning for LLMsChenxu Wang, Yilin Lyu, Zicheng Sun, Liping JingACL 2025 · 7 citations
  4. Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual LearningYitong Wang, Xue Han, Wenchun Gao, Qian Hu et al.ACL 2026
  5. Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual LearningNikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar et al.ICLR 2026 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines