Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning

Continual learning of Vision-Language Model (VLM) aims to empower models with new expertise without compromising zero-shot capabilities. However, this pursuit faces a critical “dual-forgetting” challenge: catastrophic forgetting of newly acquired classes (Incremental Knowledge Forgetting, IKF) and erosion of foundational zero-shot capabilities (Pre-trained Knowledge Forgetting, PKF). Existing evaluations often ignore PKF or assess it via confounded protocols where positive transfer on semantically similar domains creates an illusion of retention, masking severe foundational degradation. To address this, we propose the Dual-Forgetting-Aware Class-Incremental Learning (DFA-CIL) framework and the Similarity-Calibrated Retention (SCR) metric. Unlike standard averaging, SCR uses the frozen pre-trained feature space to inversely weight performance by semantic similarity, mitigating confounding gains to stress-test foundational stability. Building on this, we propose DFA-MoE, a functionally heterogeneous Parameter-Efficient Fine-Tuning (PEFT) method. DFA-MoE decouples optimization objectives by assigning a momentum-enhanced contrastive expert for feature alignment, and separate plasticity experts combining classification with auxiliary contrastive learning to adapt to new tasks while retaining historical knowledge. Extensive experiments show that our framework reveals the hidden fragility of existing methods and achieves a state-of-the-art balance in preserving both incremental and pre-trained knowledge. Our code is available at https://github.com/RL-MIND/DFA-MoE.