EMNLP Findings2025
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
Zeping Yu, Sophia Ananiadou
Abstract
Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM's language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuronlevel strategy that preserves the influence of neurons with large parameter shifts-neurons likely responsible for newly acquired visual capabilities-while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Visual Instruction Tuning cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering cites Language Models are Few-Shot Learners
Cited by top-tier papers1
Builds on21
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
Related papers
- PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMsZijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie et al.ACL Findings 2026
- Multi-Modality Expansion and Retention for LLMs through Parameter Merging and DecouplingJunlin Li, Guodong Du, Jing Li, Sim Kuan Goh et al.ACL 2025
- LMFusion: Adapting Pretrained Language Models for Multimodal GenerationWeijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang et al.NeurIPS 2025 · 134 citations
- LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-DisjointQianli Ma, Dongrui Liu, Qian Chen, Linfeng Zhang et al.ACL 2025 · 5 citations
- Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language ModelsDidi Zhu, Zhongyi Sun, Zexi Li, Tao Shen et al.ICML 2024 · 50 citations
