Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning

Instruction tuning for large language models (LLMs) can drive them to produce results consistent with human goals in specific downstream tasks. However, the process of continual instruction tuning (CIT) for LLMs may bring about the catastrophic forgetting (CF) problem, where previously learned abilities are degraded. Recent methods try to alleviate the CF problem by modifying models or replaying data, which may only remember the surface-level pattern of instructions and get confused on held-out tasks. In this paper, we propose a novel continual instruction tuning method based on Key-part Information Gain (KPIG). Our method computes the information gain on masked parts to dynamically replay data and refine the training objective, which enables LLMs to capture task-aware information relevant to the correct response and alleviate overfitting to general descriptions in instructions. In addition, we propose two metrics, P-score and V-score, to measure the generalization and instruction-following abilities of LLMs. Experiments demonstrate our method achieves superior performance on both seen and held-out tasks. 042 Although tuning a pretrained LLM with instruc-043 tion data before deployment gains wide applica-044 tion, it still faces challenges when dealing with 045 incremental data and tasks (Zhang et al., 2023c). 046 Continual learning (CL) (Biesialska et al., 2020) 047 is introduced to avoid costly retraining on all col-048 lected instances (Biesialska et al., 2020), and con-049 tinual instruction tuning (CIT) (Zhang et al., 2023c) 050 is a sub-task of it about instruction data. However, 051 catastrophic forgetting (CF) is still an unavoidable 052 problem during CIT, which refers to the forgetting 053 of previously learned tasks and the deterioration of 054 original generalization ability (Zhao et al., 2022; 055 Zeng et al., 2023b; Zhang et al., 2023c). 056 Recently, replay, architecture, and regulariza-057 tion are three main strategies to mitigate the CF 058 problem. Replay is the most prevalent strategy that 059 leverages task-specific features to replay a small 060 130 fine the training objective. 3) We propose a novel 131 evaluation metric V-score centered on instruction-132 following ability. 4) Compared to other CL base-133 lines, our method achieves state-of-the-art perfor-134 mance on public and domain datasets.