Any-SSR: How Recursive Least Squares Works in Continual Learning of Large Language Models

Fine-tuning large language models (LLMs) on data of various domains encompassing advanced language-related tasks. Such a process is essentially a continual learning procedure since it is impractical to re-access the pre-trained data in most parties. This naturally attracts catastrophic forgetting of LLMs' previously learned knowledge, such as general skills. Existing techniques either leverage previous data to replay, leading to extra computational costs, or utilize a single parameter-efficient module to learn the downstream task, constraining new knowledge absorption with interference among different tasks. To handle these issues, we propose an Analytic Subspace Routing (Any-SSR) to address these challenges. For each task, we isolate the learning within a subspace of deep layers' features via low-rank adaptation, eliminating knowledge interference between different tasks. Additionally, we propose an analytic routing mechanism to properly utilize knowledge learned in different subspaces. Our approach employs Recursive Least Squares to train a multi-task router model, allowing the router to dynamically adapt to incoming data without requiring access to historical data. Subsequently, the router effectively assigns the current task to an appropriate subspace and has a non-forgetting property of previously learned tasks with a solid theoretical guarantee. Experimental results demonstrate that our method achieves near-perfect retention of prior knowledge while seamlessly integrating new information, effectively mitigating the core limitations of existing methods. Our code is available at https://github.com/ZHUANGHP/Any-SSR.