CVPR Findings2026
Model Merging on Loss Landscapes: A Geometric Perspective
Juanwu Lu, Anand Bhaskar, Brian Axelrod, Ekaterina Tolstaya, Tristan Emrich
Abstract
Model merging offers a promising avenue for knowledge integration and parallel development without retraining. Yet, existing methods either ignore the geometry of the loss landscape or rely on intractable full-space Hessian approximations. We propose EpiMer, a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold and restricts the computation to a low-rank subspace spanned by the task vectors. With the expected Hessian as the metric, we reveal a connection between local curvature and epistemic uncertainty of the parameters. Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy, and provides a closed-form characterization of when curvatureaware merging provably outperforms flat-geometry methods. In addition, our framework unifies both curvatureaware methods and recent spectral methods as special cases of the subspace Fréchet mean with different geometric metrics. Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank, improving the across-task average accuracy and worst-task accuracy on every backbone.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- TIES-Merging: Resolving Interference When Merging Models cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
- TIES-Merging: Resolving Interference When Merging Models cites Learning Transferable Visual Models From Natural Language Supervision
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Merging Models with Fisher-Weighted Averaging
- Merging Models with Fisher-Weighted Averaging cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Learning Transferable Visual Models From Natural Language Supervision
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Merging Models with Fisher-Weighted Averaging
Builds on15
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
- TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
- Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations
Related papers
- Orthogonal Model MergingSihan Yang, Kexuan Shi, Weiyang LiuICML 2026 · 2 citations
- CAMEx: Curvature-aware Merging of ExpertsViet Dung Nguyen, Minh Nguyen Hoang, Luc Q. Nguyen, Rachel S. Y. Teo et al.ICLR 2025
- Geometric Embedding Alignment via Curvature Matching in Transfer LearningSung Moon Ko, Jaewan Lee, Sumin Lee, Soorin Yim et al.ICML 2026
- Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow MatchingLi Ju, Mayank Nautiyal, Andreas Hellander, Ekta Vats et al.ICML 2026 · 2 citations
- Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core SubspaceWenju Sun, Qingyong Li, Tiancheng Li, Yangliao Geng et al.ICML 2026
