ACL Findings2025
NeuronMerge: Merging Models via Functional Neuron Groups
Wangyun Gu, Qianghua Gao, Li-Xin Zhang, Xu Shen, Jieping Ye
Abstract
Model merging techniques like task arithmetic, which combines model parameters through weighted averaging, have proven effective. However, the success of task arithmetic relies on the linearity between model weight differences and output feature changes, which is often lacking in conventional fine-tuned models. In this work, we employ neuron description methods to analyze and classify neurons based on their functionalities. We theoretically demonstrate that grouping Multi-Layer Perceptron (MLP) neurons by functionality enhances model linearity. Building on this, we propose a neuron-based task arithmetic merging method that consistently improves performance across various tasks and model scales. Our approach is complementary to existing merging techniques, achieving superior results in merging models fine-tuned on fundamental tasks like Math, Code and Translation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TIES-Merging: Resolving Interference When Merging Models cites Linear Mode Connectivity and the Lottery Ticket Hypothesis
- TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
- When Model Merging Breaks Routing: Training-Free Calibration for MoE cites TIES-Merging: Resolving Interference When Merging Models
- When Model Merging Breaks Routing: Training-Free Calibration for MoE cites Linear Mode Connectivity and the Lottery Ticket Hypothesis
- When Model Merging Breaks Routing: Training-Free Calibration for MoE cites Merging Models with Fisher-Weighted Averaging
- When Model Merging Breaks Routing: Training-Free Calibration for MoE cites Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Merging Models with Fisher-Weighted Averaging
Cited by top-tier papers1
Builds on19
- TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
- Linear Mode Connectivity and the Lottery Ticket HypothesisJonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael CarbinICML 2020 · 750 citations
- Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations
- Llemma: An Open Language Model for MathematicsZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos et al.ICLR 2024 · 433 citations
Related papers
- Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMsRui Dai, Sile Hu, Xu Shen, Yonggang Zhang et al.ICLR 2025
- To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model MergingZitao Fang, Guodong Du, Shuyang Yu, Yifei Guo et al.EMNLP 2025 · 2 citations
- Merging Multi-Task Models via Weight-Ensembling Mixture of ExpertsAnke Tang, Li Shen, Yong Luo, Nan Yin et al.ICML 2024 · 96 citations
- Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task ArithmeticRuochen Jin, Bojian Hou, Jiancong Xiao, Weijie J. Su et al.ICLR 2025
- Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained ModelsGuillermo Ortiz-Jiménez, Alessandro Favero, Pascal FrossardNeurIPS 2023 · 272 citations
