Lune

ACL Findings2025

SeqMMR: Sequential Model Merging and LLM Routing for Enhanced Batched Sequential Knowledge Editing

Shanbao Qiao, Xuebing Liu, Akshat Gupta, Seung-Hoon Na

Abstract

Model knowledge editing enables the efficient correction of erroneous information and the continuous updating of outdated knowledge within language models. While existing research has demonstrated strong performance in single-instance or few-instance sequential editing and one-time massive editing scenarios, the batched sequential editing paradigm remains a significant challenge. The primary issue lies in the model's tendency to gradually forget previously edited knowledge and become increasingly unstable after multiple iterations of batched editing. To address these challenges, we propose SeqMMR, an enhanced framework for batched sequential knowledge editing that leverages Sequential Model Merging and a model Router. Our approach iteratively merges parameters from current batch-edited models with those of their predecessors, ensuring that newly emerging knowledge is integrated while mitigating the forgetting of previously edited knowledge. Furthermore, the model router directs queries unrelated to the edited knowledge to an unedited model backup, preventing unintended alterations in model predictions. Extensive experiments across various datasets demonstrate that our approach effectively mitigates knowledge forgetting, improves performance across all previous batches, and better preserves the model's general capabilities.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TIES-Merging: Resolving Interference When Merging Models cites Merging Models with Fisher-Weighted Averaging
  • TamEdit: Trajectory-Aware Meta-Learning for Specificity-Preserving Continual Knowledge Editing cites Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Measuring Massive Multitask Language Understanding
  • Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch cites Merging Models with Fisher-Weighted Averaging
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. TamEdit: Trajectory-Aware Meta-Learning for Specificity-Preserving Continual Knowledge EditingShiqiang Tian, Cheng Ding, Qin Chen, Jie Zhou et al.ACL 2026 · 1 citation

Builds on27

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. TIES-Merging: Resolving Interference When Merging ModelsPrateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel et al.NeurIPS 2023 · 999 citations
  3. Merging Models with Fisher-Weighted AveragingMichael Matena, Colin RaffelNeurIPS 2022 · 741 citations
  4. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations
  5. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value AdaptorsTom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim et al.NeurIPS 2023 · 349 citations

Related papers

  1. Hippocampal-like Sequential Editing for Continual Knowledge Updates in Large Language ModelsQuntian Fang, Zhen Huang, Zhiliang Tian, Minghao Hu et al.NeurIPS 2025 · 2 citations
  2. Neuron-Level Sequential Editing for Large Language ModelsHoucheng Jiang, Junfeng Fang, Tianyu Zhang, Baolong Bi et al.ACL 2025
  3. Wasserstein Distance Constraint and Parameter Sparsification for Batched and Iterative Knowledge EditingShanbao Qiao, Xuebing Liu, Seung-Hoon NaAAAI 2025
  4. Can We Continually Edit Language Models? On the Knowledge Attenuation in Sequential Model EditingQi Li, Xiaowen ChuACL Findings 2024 · 2 citations
  5. Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model EditingBhiman Kumar Baghel, Emma Jordan, Zheyuan Ryan Shi, Xiang Lorraine LiEMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines