LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, Mang Ye
Abstract
While Multimodal Large Language Models (MLLMs) excel at generalizing across modalities and tasks, effectively adapting them to specific downstream tasks while simultaneously retaining both general and specialized knowledge remains challenging. Although Low-Rank Adaptation (LoRA) is widely used to efficiently acquire specialized knowledge in MLLMs, it introduces substantial harmful redundancy during visual instruction tuning, which exacerbates the forgetting of general knowledge and degrades downstream task performance. To address this issue, we propose LoRASculpt to eliminate harmful redundant parameters, thereby harmonizing general and specialized knowledge. Specifically, under theoretical guarantees, we introduce sparse updates into LoRA to discard redundant parameters effectively. Furthermore, we propose a Conflict Mitigation Regularizer to refine the update trajectory of LoRA, mitigating knowledge conflicts with the pretrained weights. Extensive experimental results demonstrate that even at very high degree of sparsity (≤ 5%), our method simultaneously enhances generalization and downstream task performance. This confirms that our approach effectively mitigates the catastrophic forgetting issue and further promotes knowledge harmonization in MLLMs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Simple and Effective Pruning Approach for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Backdoor Cleaning without External Guidance in MLLM Fine-tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- Backdoor Cleaning without External Guidance in MLLM Fine-tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Backdoor Cleaning without External Guidance in MLLM Fine-tuning cites Learning Transferable Visual Models From Natural Language Supervision
- SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization cites Backdoor Cleaning without External Guidance in MLLM Fine-tuning
- SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- LoKI: Low-Damage Knowledge Implanting of Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models cites Backdoor Cleaning without External Guidance in MLLM Fine-tuning
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models cites SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization
Cited by top-tier papers13
- Backdoor Cleaning without External Guidance in MLLM Fine-tuningXuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi et al.NeurIPS 2025 · 39 citations
- SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy OptimizationXuankun Rong, Wenke Huang, Tingfeng Wang, Daiguo Zhou et al.CVPR 2026 · 13 citations
- Understanding LoRA as Knowledge Memory: An Empirical AnalysisSeungju Back, Dongwoo Lee, Naun Kang, Taehee Lee et al.ICML 2026 · 10 citations
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language ModelsYiyang Fang, Wenke Huang, Pei Fu, Yihao Yang et al.CVPR 2026 · 4 citations
- LoKI: Low-Damage Knowledge Implanting of Large Language ModelsRunyu Wang, Peng Ping, Zhengyu Guo, Xiaoye Zhang et al.AAAI 2026 · 3 citations
Builds on31
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
Related papers
- LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction TuningChang Che, Ziqi Wang, Pengwan Yang, Cheems Wang et al.AAAI 2026
- L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific RegularizationXiang Zhang, Rui Xie, Shikun ZhangAAAI 2026
- DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality LearningXiwei Liu, Yulong Li, Feilong Tang, Imran RazzakAAAI 2026
- Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained KnowledgePengwei Tang, Xiaolin Hu, Yong Liu, Lizhong Ding et al.AAAI 2026 · 6 citations
- Orthogonal Subspace Learning for Language Model Continual LearningXiao Wang, Tianze Chen, Qiming Ge, Han Xia et al.EMNLP Findings 2023 · 79 citations
