Lune

ACL Findings2026

PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs

Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Deyuan Chen, Zhengjie Zhao, Shi Feng, Daling Wang, Xiaocui Yang, Yifei Zhang, Hinrich Schütze

Abstract

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a trainingfree framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and latemodal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs across nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on taskrelevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Multimodal Few-Shot Learning with Frozen Language Models
  • Multimodal Few-Shot Learning with Frozen Language Models cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  5. Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami et al.NeurIPS 2021 · 1,020 citations

Related papers

  1. Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMsZeping Yu, Sophia AnaniadouEMNLP Findings 2025
  2. Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive AttentionShezheng Song, Shasha Li, Shan Zhao, Xiaopeng Li et al.CVPR 2026 · 3 citations
  3. Multi-Modality Expansion and Retention for LLMs through Parameter Merging and DecouplingJunlin Li, Guodong Du, Jing Li, Sim Kuan Goh et al.ACL 2025
  4. Bring Reason to Vision: Understanding Perception and Reasoning through Model MergingShiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu et al.ICML 2025
  5. VisionTrim: Unified Vision Token Compression for Training-Free MLLM AccelerationHanxun Yu, Wentong Li, Xuan Qu, Song Wang et al.ICLR 2026 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines