ACL Findings2025
NeuronMerge: Merging Models via Functional Neuron Groups
Wangyun Gu, Qianghua Gao, Li-Xin Zhang, Xu Shen, Jieping Ye
摘要
Model merging techniques like task arithmetic, which combines model parameters through weighted averaging, have proven effective. However, the success of task arithmetic relies on the linearity between model weight differences and output feature changes, which is often lacking in conventional fine-tuned models. In this work, we employ neuron description methods to analyze and classify neurons based on their functionalities. We theoretically demonstrate that grouping Multi-Layer Perceptron (MLP) neurons by functionality enhances model linearity. Building on this, we propose a neuron-based task arithmetic merging method that consistently improves performance across various tasks and model scales. Our approach is complementary to existing merging techniques, achieving superior results in merging models fine-tuned on fundamental tasks like Math, Code and Translation.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- TIES-Merging: Resolving Interference When Merging Models 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
- TIES-Merging: Resolving Interference When Merging Models 引用了 Merging Models with Fisher-Weighted Averaging
- When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 TIES-Merging: Resolving Interference When Merging Models
- When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
- When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Merging Models with Fisher-Weighted Averaging
- When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch 引用了 Merging Models with Fisher-Weighted Averaging
引用它的顶会 Paper1
它引用的 Paper19
- TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel 等NeurIPS 2023 · 999 次引用
- Linear Mode Connectivity and the Lottery Ticket HypothesisJonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael CarbinICML 2020 · 750 次引用
- Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 次引用
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang 等ICML 2024 · 605 次引用
- Llemma: An Open Language Model for MathematicsZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos 等ICLR 2024 · 433 次引用
相关 Paper
- Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMsRui Dai, Sile Hu, Xu Shen, Yonggang Zhang 等ICLR 2025
- To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model MergingZitao Fang, Guodong Du, Shuyang Yu, Yifei Guo 等EMNLP 2025 · 2 次引用
- Merging Multi-Task Models via Weight-Ensembling Mixture of ExpertsAnke Tang, Li Shen, Yong Luo, Nan Yin 等ICML 2024 · 96 次引用
- Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task ArithmeticRuochen Jin, Bojian Hou, Jiancong Xiao, Weijie J. Su 等ICLR 2025
- Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained ModelsGuillermo Ortiz-Jiménez, Alessandro Favero, Pascal FrossardNeurIPS 2023 · 272 次引用
