Superpose Task-specific Features for Model Merging

Model merging enables powerful capabilities in neural networks without requiring additional training.In this paper, we introduce a novel perspective on model merging by leveraging the fundamental mechanisms of neural network representation.Our approach is motivated by the linear representation hypothesis, which states that neural networks encode information through linear combinations of feature vectors.We propose a method that superposes task-specific features from individual models into a merged model.Our approach specifically targets linear transformation matrices, which are crucial for feature activation and extraction in deep networks.By formulating the merging process as a linear system, we can preserve task-specific features from individual models and create merged models that effectively maintain multi-task capabilities compared to existing methods.Extensive experiments across diverse benchmarks and models demonstrate that our method outperforms existing techniques.Code is available at https://github.com/LARS-research/STF. and task vectors (Ilharco et al., 2022;Du et al., 2024).However, these methods primarily focus on parameter-level operations and do not explicitly incorporate the fundamental working mechanisms of neural networks in their design.We argue that a principled approach to model merging should be conditioned on how deep neural networks represent and process information.Therefore, to design our merging method, we draw upon e n c .0 e n c .1 e n c .2 e n c .3 e n c .4 e n c .5 e n c .6 e n c .7 e n c .8 e n c .9 e n c .1 0 e n c .1 1 d e c .0 d e c .1 d e