EMNLP Findings2023
CITB: A Benchmark for Continual Instruction Tuning
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad
Abstract
Continual learning (CL) is a paradigm that aims to replicate the human ability to learn and accumulate knowledge continually without forgetting previous knowledge and transferring it to new tasks. Recent instruction tuning (IT) involves fine-tuning models to make them more adaptable to solving NLP tasks in general. However, it is still uncertain how instruction tuning works in the context of CL tasks. This challenging yet practical problem is formulated as Continual Instruction Tuning (CIT). In this work, we establish a CIT benchmark consisting of learning and evaluation protocols. We curate two long dialogue task streams of different types, InstrDialog and InstrDialog++, to study various CL methods systematically. Our experiments show that existing CL methods do not effectively leverage the rich natural language instructions, and fine-tuning an instruction-tuned model sequentially can yield similar or better results. We further explore different aspects that might affect the learning of CIT. We hope this benchmark will facilitate more research in this direction 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- OnEDIT: Online Editing with Decoupled Implicit Task for Large Language Models cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
- OnEDIT: Online Editing with Decoupled Implicit Task for Large Language Models cites Finetuned Language Models are Zero-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Finetuned Language Models are Zero-Shot Learners
Cited by top-tier papers8
- Learn more, but bother less: parameter efficient continual learningFuli Qiao, Mehrdad MahdaviNeurIPS 2024 · 36 citations
- Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized RehearsalJianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang et al.ACL 2024 · 13 citations
- SMoLoRa: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction TuningZiqi Wang, Chang Che, Qi Wang, Yangyang Li et al.ICCV 2025 · 4 citations
- OnEDIT: Online Editing with Decoupled Implicit Task for Large Language ModelsChae-Won Lee, Jae-Hong Lee, Ji-Hun Kang, Joon-Hyuk ChangAAAI 2026
- Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data SelectionAdyasha Maharana, Jaehong Yoon, Tianlong Chen, Mohit BansalICLR 2025
Builds on13
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
- The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi et al.EMNLP 2022 · 238 citations
- Achieving Forgetting Prevention and Knowledge Transfer in Continual LearningZixuan Ke, Bing Liu, Nianzu Ma, Hu Xu et al.NeurIPS 2021 · 167 citations
Related papers
- ConTinTin: Continual Learning from Task InstructionsWenpeng Yin, Jia Li, Caiming XiongACL 2022
- Federated Continual Instruction TuningHaiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu et al.ICCV 2025 · 2 citations
- Don't Half-listen: Capturing Key-part Information in Continual Instruction TuningYongquan He, Wenyuan Zhang, Xuancheng Huang, Peng Zhang et al.ACL 2025
- Continual Learning in Task-Oriented Dialogue SystemsAndrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon et al.EMNLP 2021 · 68 citations
- InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction TuningPrakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri et al.EMNLP 2022 · 26 citations
