Lune

ACL Findings2026

PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs

Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Deyuan Chen, Zhengjie Zhao, Shi Feng, Daling Wang, Xiaocui Yang, Yifei Zhang, Hinrich Schütze

摘要

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a trainingfree framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and latemodal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs across nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on taskrelevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
  • Visual Instruction Tuning 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning 引用了 Multimodal Few-Shot Learning with Frozen Language Models
  • Multimodal Few-Shot Learning with Frozen Language Models 引用了 Learning Transferable Visual Models From Natural Language Supervision
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper14

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 47,906 次引用
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech 等NeurIPS 2022 · 6,707 次引用
  4. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong 等NeurIPS 2023 · 4,013 次引用
  5. Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami 等NeurIPS 2021 · 1,020 次引用

相关 Paper

  1. Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMsZeping Yu, Sophia AnaniadouEMNLP Findings 2025
  2. Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive AttentionShezheng Song, Shasha Li, Shan Zhao, Xiaopeng Li 等CVPR 2026 · 3 次引用
  3. Multi-Modality Expansion and Retention for LLMs through Parameter Merging and DecouplingJunlin Li, Guodong Du, Jing Li, Sim Kuan Goh 等ACL 2025
  4. Bring Reason to Vision: Understanding Perception and Reasoning through Model MergingShiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu 等ICML 2025
  5. VisionTrim: Unified Vision Token Compression for Training-Free MLLM AccelerationHanxun Yu, Wentong Li, Xuan Qu, Song Wang 等ICLR 2026 · 17 次引用
黄昏的海面,两侧是细线勾勒的悬崖