Lune

CVPR Findings2026

Model Merging on Loss Landscapes: A Geometric Perspective

Juanwu Lu, Anand Bhaskar, Brian Axelrod, Ekaterina Tolstaya, Tristan Emrich

Abstract

Model merging offers a promising avenue for knowledge integration and parallel development without retraining. Yet, existing methods either ignore the geometry of the loss landscape or rely on intractable full-space Hessian approximations. We propose EpiMer, a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold and restricts the computation to a low-rank subspace spanned by the task vectors. With the expected Hessian as the metric, we reveal a connection between local curvature and epistemic uncertainty of the parameters. Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy, and provides a closed-form characterization of when curvatureaware merging provably outperforms flat-geometry methods. In addition, our framework unifies both curvatureaware methods and recent spectral methods as special cases of the subspace Fréchet mean with different geometric metrics. Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank, improving the across-task average accuracy and worst-task accuracy on every backbone.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TIES-Merging: Resolving Interference When Merging Models cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
  • TIES-Merging: Resolving Interference When Merging Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Merging Models with Fisher-Weighted Averaging
  • Merging Models with Fisher-Weighted Averaging cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Learning Transferable Visual Models From Natural Language Supervision
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Merging Models with Fisher-Weighted Averaging
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  3. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
  4. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations
  5. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations

Related papers

  1. Orthogonal Model MergingSihan Yang, Kexuan Shi, Weiyang LiuICML 2026 · 2 citations
  2. CAMEx: Curvature-aware Merging of ExpertsViet Dung Nguyen, Minh Nguyen Hoang, Luc Q. Nguyen, Rachel S. Y. Teo et al.ICLR 2025
  3. Geometric Embedding Alignment via Curvature Matching in Transfer LearningSung Moon Ko, Jaewan Lee, Sumin Lee, Soorin Yim et al.ICML 2026
  4. Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow MatchingLi Ju, Mayank Nautiyal, Andreas Hellander, Ekta Vats et al.ICML 2026 · 2 citations
  5. Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core SubspaceWenju Sun, Qingyong Li, Tiancheng Li, Yangliao Geng et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines