DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models

Among the recently emerging knowledgeediting methods, in-context knowledge editing (IKE) (Zheng et al., 2023) has shown respectable abilities in knowledge editing in terms of generalization and specificity. Noting the promising advantages but unexplored issues of IKE, we propose DistillMIKE as a novel extension of IKE, i.e., editing distillation of "Massive" In-context Knowledge Editing in large language models (LLMs), primarily consisting of two expansions: 1) Massive incontext knowledge editing (MIKE), which extends IKE to a massive editing task, aims to inject not a single edit but a set of massive edits into LLMs. To preserve specificity, our key novel extension is a "selective" retrieval augmentation, where the retrieval-augmented IKE is only applied to "in-scope" examples, whereas the unedited model without IKE is employed for "out-of-scope" ones. 2) Editing distillation of MIKE using low-rank adaptation (LoRA), which distills the editing abilities of MIKE to the parameters of LLMs in a manner that eliminates the need for lengthy in-context demonstrations, thereby removing the computational overhead encountered at the inference time. The experimental results on the zsRE and CounterFact datasets demonstrate that MIKE shows state-of-the-art performance, whereas DistilMIKE shows comparable performance to MIKE. Our code is available at https://github.com/JoveReCode/ DistillMIKE.git .