Lune

EMNLP Findings2025

Exploring Model Kinship for Merging Large Language Models

Yedi Hu, Yunzhi Yao, Ningyu Zhang, Huajun Chen, Shumin Deng

Abstract

Model merging has emerged as a key technique for enhancing the capabilities and efficiency of Large Language Models (LLMs). The opensource community has driven model evolution by iteratively merging existing models, yet a principled understanding of the gains and underlying factors in model merging remains limited. In this work, we study model evolution through iterative merging, drawing an analogy to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. Through comprehensive empirical analysis, we show that model kinship is closely linked to the performance improvements achieved by merging, providing a useful criterion for selecting candidate models. Building on this insight, we propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can improve benchmark performance. Specifically, we discover that incorporating model kinship as a guiding criterion enables continuous merging while mitigating performance degradation caused by local optima, thereby facilitating more effective model evolution 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TIES-Merging: Resolving Interference When Merging Models cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • TIES-Merging: Resolving Interference When Merging Models cites Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Linear Mode Connectivity and the Lottery Ticket Hypothesis
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Merging Models with Fisher-Weighted Averaging
  • Merging Models with Fisher-Weighted Averaging cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on24

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  3. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
  4. Linear Mode Connectivity and the Lottery Ticket HypothesisJonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael CarbinICML 2020 · 750 citations
  5. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations

Related papers

  1. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in BenchmarksNicolas Yax, Pierre-Yves Oudeyer, Stefano PalminteriICLR 2025
  2. Nature-Inspired Population-Based Evolution of Large Language ModelsYiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng et al.ACL 2026 · 12 citations
  3. Do Not Merge My Model! Safeguarding Open-Source LLMs Against Unauthorized Model MergingQinfeng Li, Miao Pan, Jintao Chen, Fu Teng et al.AAAI 2026 · 1 citation
  4. Training-free LLM Merging for Multi-task LearningZichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang et al.ACL 2025
  5. Activation-Informed Merging of Large Language ModelsAmin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava et al.NeurIPS 2025 · 25 citations
Dusk over the sea between two cliffs drawn in fine vertical lines