Lune

ACL Findings2025

NeuronMerge: Merging Models via Functional Neuron Groups

Wangyun Gu, Qianghua Gao, Li-Xin Zhang, Xu Shen, Jieping Ye

摘要

Model merging techniques like task arithmetic, which combines model parameters through weighted averaging, have proven effective. However, the success of task arithmetic relies on the linearity between model weight differences and output feature changes, which is often lacking in conventional fine-tuned models. In this work, we employ neuron description methods to analyze and classify neurons based on their functionalities. We theoretically demonstrate that grouping Multi-Layer Perceptron (MLP) neurons by functionality enhances model linearity. Building on this, we propose a neuron-based task arithmetic merging method that consistently improves performance across various tasks and model scales. Our approach is complementary to existing merging techniques, achieving superior results in merging models fine-tuned on fundamental tasks like Math, Code and Translation.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • TIES-Merging: Resolving Interference When Merging Models 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • TIES-Merging: Resolving Interference When Merging Models 引用了 Merging Models with Fisher-Weighted Averaging
  • When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 TIES-Merging: Resolving Interference When Merging Models
  • When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Merging Models with Fisher-Weighted Averaging
  • When Model Merging Breaks Routing: Training-Free Calibration for MoE 引用了 Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch 引用了 Merging Models with Fisher-Weighted Averaging
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper1

  1. When Model Merging Breaks Routing: Training-Free Calibration for MoECanbin Huang, Tianyuan Shi, Xiaojun Quan, Jingang Wang 等ICML 2026

它引用的 Paper19

  1. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel 等NeurIPS 2023 · 999 次引用
  2. Linear Mode Connectivity and the Lottery Ticket HypothesisJonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael CarbinICML 2020 · 750 次引用
  3. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 次引用
  4. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang 等ICML 2024 · 605 次引用
  5. Llemma: An Open Language Model for MathematicsZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos 等ICLR 2024 · 433 次引用

相关 Paper

  1. Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMsRui Dai, Sile Hu, Xu Shen, Yonggang Zhang 等ICLR 2025
  2. To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model MergingZitao Fang, Guodong Du, Shuyang Yu, Yifei Guo 等EMNLP 2025 · 2 次引用
  3. Merging Multi-Task Models via Weight-Ensembling Mixture of ExpertsAnke Tang, Li Shen, Yong Luo, Nan Yin 等ICML 2024 · 96 次引用
  4. Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task ArithmeticRuochen Jin, Bojian Hou, Jiancong Xiao, Weijie J. Su 等ICLR 2025
  5. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained ModelsGuillermo Ortiz-Jiménez, Alessandro Favero, Pascal FrossardNeurIPS 2023 · 272 次引用
黄昏的海面,两侧是细线勾勒的悬崖