EMNLP Findings2024
Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization
Mingyang Wang, Lukas Lange, Heike Adel, Jannik Strötgen, Hinrich Schütze
Abstract
To ensure large language models contain up-todate knowledge, they need to be updated regularly. However, model editing is challenging as it might also affect knowledge that is unrelated to the new data. State-of-the-art methods identify parameters associated with specific knowledge and then modify them via direct weight updates. However, these locate-and-edit methods suffer from heavy computational overhead and lack theoretical validation. In contrast, directly fine-tuning the model on requested edits affects the model's behavior on unrelated knowledge, and significantly damages the model's generation fluency and consistency. To address these challenges, we propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization. Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-theart methods while maintaining generation quality and reducing computational overhead.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models cites Memory-Based Model Editing at Scale
- Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models cites Fast Model Editing at Scale
- Fast Model Editing at Scale cites LoRA: Low-Rank Adaptation of Large Language Models
- Editing Large Language Models: Problems, Methods, and Opportunities cites Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
Builds on12
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
- Memory-Based Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning et al.ICML 2022 · 520 citations
- Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language ModelsPeter Hase, Mohit Bansal, Been Kim, Asma GhandehariounNeurIPS 2023 · 307 citations
- Editing Large Language Models: Problems, Methods, and OpportunitiesYunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng et al.EMNLP 2023 · 83 citations
Related papers
- Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model EditingBhiman Kumar Baghel, Emma Jordan, Zheyuan Ryan Shi, Xiang Lorraine LiEMNLP Findings 2025 · 1 citation
- Model Editing Harms General Abilities of Large Language Models: Regularization to the RescueJia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu et al.EMNLP 2024 · 8 citations
- Lifelong Knowledge Editing requires Better RegularizationAkshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed M. Alaa et al.EMNLP Findings 2025 · 1 citation
- STEAM: A Semantic-Level Knowledge Editing Framework for Large Language ModelsGeunyeong Jeong, Juoh Sun, Seonghee Lee, Harksoo KimEMNLP Findings 2025 · 1 citation
- Should We Really Edit Language Models? On the Evaluation of Edited Language ModelsQi Li, Xiang Liu, Zhenheng Tang, Peijie Dong et al.NeurIPS 2024 · 25 citations
