Lune

AAAI2025Top-tier venue

Wasserstein Distance Constraint and Parameter Sparsification for Batched and Iterative Knowledge Editing

Shanbao Qiao, Xuebing Liu, Seung-Hoon Na

Abstract

Model knowledge editing has become a widely researched topic because it enables the efficient and rapid injection of new knowledge into language models, as well as the correction of erroneous or outdated knowledge. Existing model knowledge editing methods are typically classified into two categories: single-instance sequential editing and massive one-time editing. However, the batched and iterative editing method is better aligned with model updating patterns in practical applications. In this work, we examine the performance of parameter-update-based models on a novel batched iterative editing benchmark. Our findings indicate that as the number of editing iterations increases, the accumulation of updated parameters leads to a greater shift in the model's parameter distribution, making it harder to maintain both the editing performance and the stability of the model. To address this degradation issue, we propose two methods: the Wasserstein distance constraint and the parameter update sparsification, where the Wasserstein distance constraint optimizes the transition of parameter distribution before and after editing, and the parameter update sparsification significantly reduces the number of updated parameters, thereby alleviating instability in the parameter distribution caused by their accumulation over iterations. Our methods are generally applicable to various parameter-update-based knowledge editing models. Experiments on the zsRE and CounterFact datasets demonstrate that our methods improve editing performance and enhance the stability of batched iterative editing in the later stages across different models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Mind the Gap: Assessing Temporal Generalization in Neural Language Models cites Language Models are Few-Shot Learners
  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Language Models are Few-Shot Learners
  • StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
  • StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value AdaptorsTom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim et al.NeurIPS 2023 · 349 citations
  3. Mind the Gap: Assessing Temporal Generalization in Neural Language ModelsAngeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal et al.NeurIPS 2021 · 315 citations
  4. PMET: Precise Model Editing in a TransformerXiaopeng Li, Shasha Li, Shezheng Song, Jing Yang et al.AAAI 2024 · 208 citations
  5. StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering ModelsAdam Liska, Tomás Kociský, Elena Gribovskaya, Tayfun Terzi et al.ICML 2022 · 129 citations

Related papers

  1. Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model EditingBhiman Kumar Baghel, Emma Jordan, Zheyuan Ryan Shi, Xiang Lorraine LiEMNLP Findings 2025 · 1 citation
  2. SeqMMR: Sequential Model Merging and LLM Routing for Enhanced Batched Sequential Knowledge EditingShanbao Qiao, Xuebing Liu, Akshat Gupta, Seung-Hoon NaACL Findings 2025
  3. History Matters: Temporal Knowledge Editing in Large Language ModelXunjian Yin, Jin Jiang, Liming Yang, Xiaojun WanAAAI 2024 · 18 citations
  4. Can We Continually Edit Language Models? On the Knowledge Attenuation in Sequential Model EditingQi Li, Xiaowen ChuACL Findings 2024 · 2 citations
  5. AdaEdit: Advancing Continuous Knowledge Editing For Large Language ModelsQi Li, Xiaowen ChuACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines