Continual Model Merging without Data: Dual Projections for Balancing Stability and Plasticity

Model merging integrates multiple expert models with diverse capabilities into a unified framework, facilitating collaborative learning. However, most existing methods assume simultaneous access to all models, which is often impractical in real-world scenarios where models are received sequentially. While some studies have investigated continual model merging (CMM)-which involves sequentially merging multiple models-the challenge of balancing prior knowledge (stability) and incorporating new tasks (plasticity) remains unresolved. This paper, for the first time, formally defines the stability and plasticity of CMM from the perspective of orthogonal projection. Subsequently, we analyze the relationships among the spaces spanned by task data, historical gradients, and accumulated gradients. Building on this, we propose a data-free Dual Orthogonal Projection (DOP) method, which eliminates data dependence and mitigates interference between the merged model and models for old and new tasks by projecting their parameter differences onto their respective approximate data spaces. Finally, to solve potential conflicts between stability and plasticity, we reformulate DOP as a multi-objective optimization problem and employ a multi-gradient descent algorithm to obtain a Pareto-optimal solution. Extensive experiments across multiple architectures and task configurations validate that our approach significantly outperforms state-of-the-art CMM methods. * This work was completed during Enneng Yang's study and employment at institutions 1, 2, 5. † Corresponding authors. 39th Conference on Neural Information Processing Systems (NeurIPS 2025).

• We formalize the two key optimization objectives of CMM, namely stability and plasticity, within the framework of orthogonal projection theory, and we further examine the relationships among the subspaces spanned by task data, gradients, and accumulated gradients.

• We propose a data-free Dual Orthogonal Projection (DOP) method for CMM, recasting it as a multi-objective optimization problem to obtain a Pareto-optimal solution, ensuring an effective trade-off between stability and plasticity.

• We conduct extensive CMM experiments on four architectures (covering vision and language models) with varying task numbers, and show that our method consistently outperforms state-ofthe-art (SOTA) CMM approaches on both old and new tasks.