Lune

EMNLP Findings2025

Exploring Model Kinship for Merging Large Language Models

Yedi Hu, Yunzhi Yao, Ningyu Zhang, Huajun Chen, Shumin Deng

摘要

Model merging has emerged as a key technique for enhancing the capabilities and efficiency of Large Language Models (LLMs). The opensource community has driven model evolution by iteratively merging existing models, yet a principled understanding of the gains and underlying factors in model merging remains limited. In this work, we study model evolution through iterative merging, drawing an analogy to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. Through comprehensive empirical analysis, we show that model kinship is closely linked to the performance improvements achieved by merging, providing a useful criterion for selecting candidate models. Building on this insight, we propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can improve benchmark performance. Specifically, we discover that incorporating model kinship as a guiding criterion enables continuous merging while mitigating performance degradation caused by local optima, thereby facilitating more effective model evolution 1 .

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • TIES-Merging: Resolving Interference When Merging Models 引用了 Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • TIES-Merging: Resolving Interference When Merging Models 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • TIES-Merging: Resolving Interference When Merging Models 引用了 Merging Models with Fisher-Weighted Averaging
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time 引用了 Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time 引用了 Merging Models with Fisher-Weighted Averaging
  • Merging Models with Fisher-Weighted Averaging 引用了 Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper24

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 24,707 次引用
  2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs 等ICML 2022 · 1,464 次引用
  3. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel 等NeurIPS 2023 · 999 次引用
  4. Linear Mode Connectivity and the Lottery Ticket HypothesisJonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael CarbinICML 2020 · 750 次引用
  5. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 次引用

相关 Paper

  1. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in BenchmarksNicolas Yax, Pierre-Yves Oudeyer, Stefano PalminteriICLR 2025
  2. Nature-Inspired Population-Based Evolution of Large Language ModelsYiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng 等ACL 2026 · 12 次引用
  3. Do Not Merge My Model! Safeguarding Open-Source LLMs Against Unauthorized Model MergingQinfeng Li, Miao Pan, Jintao Chen, Fu Teng 等AAAI 2026 · 1 次引用
  4. Training-free LLM Merging for Multi-task LearningZichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang 等ACL 2025
  5. Activation-Informed Merging of Large Language ModelsAmin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava 等NeurIPS 2025 · 25 次引用
黄昏的海面,两侧是细线勾勒的悬崖