Lune

ICML2025Top-tier venue

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging

Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He

Abstract

Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation. Our code is publicly available at: https://github.com/shiqichen17/VLM Merging .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time cites Learning Transferable Visual Models From Natural Language Supervision
  • Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training cites Learning Transferable Visual Models From Natural Language Supervision
  • VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models cites Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  • Are We on the Right Way for Evaluating Large Vision-Language Models? cites Learning Transferable Visual Models From Natural Language Supervision
  • CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs cites Sigmoid Loss for Language Image Pre-Training
  • CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. Mechanistic Detection and Mitigation of Hallucination in Large Reasoning ModelsZhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang et al.ICLR 2026 · 30 citations
  2. Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingJunlin Han, Shengbang Tong, David Fan, Yufan Ren et al.ICLR 2026 · 25 citations
  3. MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action AgentYuxia Fu, Zhizhen Zhang, Yuqi Zhang, Zijian Wang et al.CVPR 2026 · 21 citations
  4. VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding ModelsLingjie Jiang, Shaohan Huang, Xun Wu, Yixia Li et al.ICLR 2026 · 15 citations
  5. CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMsJiwan Kim, Kibum Kim, Sangwoo Seo, Chanyoung ParkICLR 2026 · 13 citations

Builds on22

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
  3. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
  4. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  5. Are We on the Right Way for Evaluating Large Vision-Language Models?Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang et al.NeurIPS 2024 · 1,029 citations

Related papers

  1. Beyond Layer-Wise Merging: Chain-of-Merging for Vision-Language ModelsXinyu Zhang, Yuxuan Dong, Lingling Zhang, Chengyou Jia et al.CVPR 2026
  2. FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language ModelsChenyu Huang, Peng Ye, Xudong Tan, Jinhan Mu et al.ICML 2026
  3. Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem SolvingZixian Guo, Ming Liu, Qilong Wang, Zhilong Ji et al.ICCV 2025 · 1 citation
  4. ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and WisdomJingqi Zhou, Sheng Wang, Jingwei Dong, Kai Liu et al.EMNLP 2025
  5. Prism: A Framework for Decoupling and Assessing the Capabilities of VLMsYuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang et al.NeurIPS 2024 · 49 citations
Dusk over the sea between two cliffs drawn in fine vertical lines