Lune

ACL Findings2024

Can We Continually Edit Language Models? On the Knowledge Attenuation in Sequential Model Editing

Qi Li, Xiaowen Chu

Abstract

Model editing has become a promising approach for precisely and effectively updating knowledge within language models. In this paper, we investigate knowledge attenuation, in which the retention of updated knowledge within the language model decreases as the number of edits increases after sequential editing. Through empirical study, we discovered that existing editing methods generally suffer from knowledge attenuation. We attribute this phenomenon to two aspects: (1) redundant parameters interference and (2) update weight disentanglement. To this end, we propose the AdaPLE method. It not only mitigates the knowledge attenuation issue but also improves the performance on existing benchmarks. To the best of our knowledge, we are the first to investigate the cause and mitigation of knowledge attenuation in sequential LLM editing.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
  • TIES-Merging: Resolving Interference When Merging Models cites Robust fine-tuning of zero-shot models
  • Hippocampal-like Sequential Editing for Continual Knowledge Updates in Large Language Models cites Memory-Based Model Editing at Scale
  • The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models cites Memory-Based Model Editing at Scale
  • TamEdit: Trajectory-Aware Meta-Learning for Specificity-Preserving Continual Knowledge Editing cites GeoEdit: Geometric Knowledge Editing for Large Language Models
  • Merging Models with Fisher-Weighted Averaging cites Robust fine-tuning of zero-shot models
  • Robust fine-tuning of zero-shot models cites Merging Models with Fisher-Weighted Averaging
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Merging Models with Fisher-Weighted Averaging
  • Should We Really Edit Language Models? On the Evaluation of Edited Language Models cites Memory-Based Model Editing at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Should We Really Edit Language Models? On the Evaluation of Edited Language ModelsQi Li, Xiang Liu, Zhenheng Tang, Peijie Dong et al.NeurIPS 2024 · 25 citations
  2. Hippocampal-like Sequential Editing for Continual Knowledge Updates in Large Language ModelsQuntian Fang, Zhen Huang, Zhiliang Tian, Minghao Hu et al.NeurIPS 2025 · 2 citations
  3. TamEdit: Trajectory-Aware Meta-Learning for Specificity-Preserving Continual Knowledge EditingShiqiang Tian, Cheng Ding, Qin Chen, Jie Zhou et al.ACL 2026 · 1 citation
  4. The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language ModelsZheng Wang, Kaixuan Zhang, Wanfang Chen, Jingwen Zhang et al.ICML 2026
  5. GeoEdit: Geometric Knowledge Editing for Large Language ModelsYujie Feng, Li-Ming Zhan, Zexin Lu, Yongxin Xu et al.EMNLP 2025

Builds on26

  1. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
  2. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations
  3. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations
  4. Memory-Based Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning et al.ICML 2022 · 520 citations
  5. Robust fine-tuning of zero-shot modelsMitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li et al.CVPR 2022 · 364 citations

Related papers

  1. AdaEdit: Advancing Continuous Knowledge Editing For Large Language ModelsQi Li, Xiaowen ChuACL 2025
  2. Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model EditingBhiman Kumar Baghel, Emma Jordan, Zheyuan Ryan Shi, Xiang Lorraine LiEMNLP Findings 2025 · 1 citation
  3. Disentangling Knowledge Representations for Large Language Model EditingMengqi Zhang, Zisheng Zhou, Xiaotian Ye, Qiang Liu et al.ICLR 2026 · 6 citations
  4. SeqMMR: Sequential Model Merging and LLM Routing for Enhanced Batched Sequential Knowledge EditingShanbao Qiao, Xuebing Liu, Akshat Gupta, Seung-Hoon NaACL Findings 2025
  5. On the Superimposed Noise Accumulation Problem in Sequential Knowledge Editing of Large Language ModelsDing Cao, Yuchen Cai, Yuqing Huang, Xuesong He et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines