Lune

EMNLP Findings2023

Visually Grounded Continual Language Learning with Selective Specialization

Kyra Ahrens, Lennart Bengtson, Jae Hee Lee, Stefan Wermter

Abstract

A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines. 1 Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Forget-free Continual Learning with Winning Subnetworks cites Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
  • Continual Learning via Local Module Composition cites Efficient Continual Learning with Modular Networks and Task-Driven Priors
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Continual Learning with Node-Importance based Adaptive Group Sparse RegularizationSangwon Jung, Hongjoon Ahn, Sungmin Cha, Taesup MoonNeurIPS 2020 · 176 citations
  2. Forget-free Continual Learning with Winning SubnetworksHaeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon et al.ICML 2022 · 159 citations
  3. Efficient Continual Learning with Modular Networks and Task-Driven PriorsTom Veniat, Ludovic Denoyer, Marc'Aurelio RanzatoICLR 2021 · 110 citations
  4. Continual Learning via Local Module CompositionOleksiy Ostapenko, Pau Rodríguez, Massimo Caccia, Laurent CharlinNeurIPS 2021 · 98 citations
  5. SparCL: Sparse Continual Learning on the EdgeZifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan et al.NeurIPS 2022 · 97 citations

Related papers

  1. Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language ModelsHyundong Jin, Hyung Jin Chang, Eunwoo KimICCV 2025 · 2 citations
  2. VQACL: A Novel Visual Question Answering Continual Learning SettingXi Zhang, Feifei Zhang, Changsheng XuCVPR 2023
  3. Re-evaluating Continual VQA: Toward Fair and Robust Evaluation for Multimodal Continual LearningZijian Gao, Zicheng Sun, Xingxing Zhang, Kele Xu et al.CVPR 2026
  4. On The Specialization of Neural ModulesDevon Jarvis, Richard Klein, Benjamin Rosman, Andrew M. SaxeICLR 2023 · 4 citations
  5. CoPE: Continual Probe-guided Expansion for Large Vision-Language ModelsZiqin Wang, Hengyuan Zhao, Qixin Sun, Kaiyou Song et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines