Lune

ACL Findings2025

Progressive LoRA for Multimodal Continual Instruction Tuning

Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, Chenhui Chu

Abstract

Multimodal Continual Instruction Tuning (MCIT) empowers Multimodal Large Language Models (MLLMs) to adapt to everevolving requirements without continuous costly retraining. However, MCIT faces challenges in mitigating Catastrophic Forgetting (CF) and enhancing Knowledge Transfer (KT). Existing works combine Mixture-of-Expert (MoE) and LoRA to address these. However, using a fixed number of shared LoRA blocks across tasks can lead to the overwriting of acquired knowledge, making MLLMs harder to handle CF and KT. Therefore, we propose the Progressive LoRA framework (ProgLoRA), which contains a progressive LoRA pool and trains a new LoRA block for each incremental task to reduce knowledge interference. Specifically, ProgLoRA has two key mechanisms: task-aware allocation for effectively leveraging acquired knowledge at current task and task recall for realigning the model with learned tasks. Additionally, considering different application scenarios, we design a static ProgLoRA for the more idealized basic setting and a dynamic ProgLoRA for the more realistic challenging setting. Experiments on the latest MCIT benchmark demonstrate that ProgLoRA outperforms existing approaches. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • NExT-GPT: Any-to-Any Multimodal LLM cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning cites PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning
  • From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code GenerationXuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen et al.ACL 2025 · 62 citations
  2. On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language ModelsChongyang Zhao, Mingsong Li, Haodong Lu, Dong GongCVPR 2026 · 3 citations
  3. SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction TuningZhen-Hao Xie Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye et al.ICML 2026
  4. PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction TuningWeicheng Meng, Jingyang Qiao, Zhizhong Zhang, Shaohui Liu et al.ICLR 2026
  5. From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction TuningPei Chen, Xilai Wang, Qixu Shi, Zejian Li et al.ACL 2026

Builds on15

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
  5. NExT-GPT: Any-to-Any Multimodal LLMShengqiong Wu, Hao Fei, Leigang Qu, Wei Ji et al.ICML 2024 · 786 citations

Related papers

  1. LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction TuningChang Che, Ziqi Wang, Pengwan Yang, Cheems Wang et al.AAAI 2026
  2. Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction TuningChendi Ge, Xin Wang, Zeyang Zhang, Hong Chen et al.ICML 2025
  3. Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual LearningYitong Wang, Xue Han, Wenchun Gao, Qian Hu et al.ACL 2026
  4. SMoLoRa: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction TuningZiqi Wang, Chang Che, Qi Wang, Yangyang Li et al.ICCV 2025 · 4 citations
  5. Grow-on-Demand: Sparse and Adaptive Expert Expansion for Continual Instruction TuningYing Zhang, Xingyue Guo, Yu Zhao, Xuhui Sui et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines