EMNLP Findings2023
Visually Grounded Continual Language Learning with Selective Specialization
Kyra Ahrens, Lennart Bengtson, Jae Hee Lee, Stefan Wermter
Abstract
A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines. 1 Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Forget-free Continual Learning with Winning Subnetworks cites Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
- Continual Learning via Local Module Composition cites Efficient Continual Learning with Modular Networks and Task-Driven Priors
Builds on15
- Continual Learning with Node-Importance based Adaptive Group Sparse RegularizationSangwon Jung, Hongjoon Ahn, Sungmin Cha, Taesup MoonNeurIPS 2020 · 176 citations
- Forget-free Continual Learning with Winning SubnetworksHaeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon et al.ICML 2022 · 159 citations
- Efficient Continual Learning with Modular Networks and Task-Driven PriorsTom Veniat, Ludovic Denoyer, Marc'Aurelio RanzatoICLR 2021 · 110 citations
- Continual Learning via Local Module CompositionOleksiy Ostapenko, Pau Rodríguez, Massimo Caccia, Laurent CharlinNeurIPS 2021 · 98 citations
- SparCL: Sparse Continual Learning on the EdgeZifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan et al.NeurIPS 2022 · 97 citations
Related papers
- Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language ModelsHyundong Jin, Hyung Jin Chang, Eunwoo KimICCV 2025 · 2 citations
- VQACL: A Novel Visual Question Answering Continual Learning SettingXi Zhang, Feifei Zhang, Changsheng XuCVPR 2023
- Re-evaluating Continual VQA: Toward Fair and Robust Evaluation for Multimodal Continual LearningZijian Gao, Zicheng Sun, Xingxing Zhang, Kele Xu et al.CVPR 2026
- On The Specialization of Neural ModulesDevon Jarvis, Richard Klein, Benjamin Rosman, Andrew M. SaxeICLR 2023 · 4 citations
- CoPE: Continual Probe-guided Expansion for Large Vision-Language ModelsZiqin Wang, Hengyuan Zhao, Qixin Sun, Kaiyou Song et al.ICML 2026
