Lune

NeurIPS2025Top-tier venue

Continuous Subspace Optimization for Continual Learning

Quan Cheng, Yuanyu Wan, Lingyu Wu, Chenping Hou, Lijun Zhang

Abstract

Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • KeepLoRA: Continual Learning with Residual Gradient Adaptation cites LoRA: Low-Rank Adaptation of Large Language Models
  • KeepLoRA: Continual Learning with Residual Gradient Adaptation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • KeepLoRA: Continual Learning with Residual Gradient Adaptation cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning cites Emerging Properties in Self-Supervised Vision Transformers
  • Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning cites Moment Matching for Multi-Source Domain Adaptation
  • MoCL: Metabolic Optimization for Curvature-Aware Continual Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • MoCL: Metabolic Optimization for Curvature-Aware Continual Learning cites Moment Matching for Multi-Source Domain Adaptation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. KeepLoRA: Continual Learning with Residual Gradient AdaptationMao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan et al.ICLR 2026 · 23 citations
  2. FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual LearningYujie Feng, Hao Wang, Jian Li, Xu Chu et al.ACL 2026 · 3 citations
  3. Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual LearningLingfeng He, De Cheng, Huaijie Wang, Xi Yang et al.ICML 2026
  4. MoCL: Metabolic Optimization for Curvature-Aware Continual LearningJiajun Lai, Qi Liu, Shijie Li, Huaiguang JiangICML 2026

Builds on23

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
  4. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations
  5. Moment Matching for Multi-Source Domain AdaptationXingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang et al.ICCV 2019 · 2,239 citations

Related papers

  1. PLAN: Proactive Low-Rank Allocation for Continual LearningXiequn Wang, Zhan Zhuang, Yu ZhangICCV 2025 · 5 citations
  2. Orthogonal Subspace Learning for Language Model Continual LearningXiao Wang, Tianze Chen, Qiming Ge, Han Xia et al.EMNLP Findings 2023 · 79 citations
  3. SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space SplittingHaomiao Qiu, Miao Zhang, Ziyue Qiao, Weili Guan et al.ICLR 2026 · 10 citations
  4. SLoRA: Balancing Plasticity and Forgetting in Large Language Models for Continual LearningLina Yang, Yusheng Liao, Yanfeng Wang, Yu WangACL 2026
  5. Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual LearningNikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar et al.ICLR 2026 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines