Lune

EMNLP Findings2023

CITB: A Benchmark for Continual Instruction Tuning

Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad

Abstract

Continual learning (CL) is a paradigm that aims to replicate the human ability to learn and accumulate knowledge continually without forgetting previous knowledge and transferring it to new tasks. Recent instruction tuning (IT) involves fine-tuning models to make them more adaptable to solving NLP tasks in general. However, it is still uncertain how instruction tuning works in the context of CL tasks. This challenging yet practical problem is formulated as Continual Instruction Tuning (CIT). In this work, we establish a CIT benchmark consisting of learning and evaluation protocols. We curate two long dialogue task streams of different types, InstrDialog and InstrDialog++, to study various CL methods systematically. Our experiments show that existing CL methods do not effectively leverage the rich natural language instructions, and fine-tuning an instruction-tuned model sequentially can yield similar or better results. We further explore different aspects that might affect the learning of CIT. We hope this benchmark will facilitate more research in this direction 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

8 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OnEDIT: Online Editing with Decoupled Implicit Task for Large Language Models cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
  • OnEDIT: Online Editing with Decoupled Implicit Task for Large Language Models cites Finetuned Language Models are Zero-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers8

  1. Learn more, but bother less: parameter efficient continual learningFuli Qiao, Mehrdad MahdaviNeurIPS 2024 · 36 citations
  2. Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized RehearsalJianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang et al.ACL 2024 · 13 citations
  3. SMoLoRa: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction TuningZiqi Wang, Chang Che, Qi Wang, Yangyang Li et al.ICCV 2025 · 4 citations
  4. OnEDIT: Online Editing with Decoupled Implicit Task for Large Language ModelsChae-Won Lee, Jae-Hong Lee, Ji-Hun Kang, Joon-Hyuk ChangAAAI 2026
  5. Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data SelectionAdyasha Maharana, Jaehong Yoon, Tianlong Chen, Mohit BansalICLR 2025

Builds on13

  1. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  2. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  3. The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
  4. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi et al.EMNLP 2022 · 238 citations
  5. Achieving Forgetting Prevention and Knowledge Transfer in Continual LearningZixuan Ke, Bing Liu, Nianzu Ma, Hu Xu et al.NeurIPS 2021 · 167 citations

Related papers

  1. ConTinTin: Continual Learning from Task InstructionsWenpeng Yin, Jia Li, Caiming XiongACL 2022
  2. Federated Continual Instruction TuningHaiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu et al.ICCV 2025 · 2 citations
  3. Don't Half-listen: Capturing Key-part Information in Continual Instruction TuningYongquan He, Wenyuan Zhang, Xuancheng Huang, Peng Zhang et al.ACL 2025
  4. Continual Learning in Task-Oriented Dialogue SystemsAndrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon et al.EMNLP 2021 · 68 citations
  5. InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction TuningPrakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri et al.EMNLP 2022 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines