Lune

ACL Findings2026

One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them

Ali Kholmovaia, Paul Youssef, Nandi Schoots, Christin Seifert

Abstract

Knowledge editing methods such as ROME and MEMIT update factual associations in transformer models by modifying MLP weights. While evaluated mainly by output behavior, their internal mechanism remains underexplored. We investigate whether edits rely on a common mechanism, regardless of which fact is modified. Despite fact-specific weight changes, we argue that ROME and MEMIT target the same subset of weights that are critical for maintaining edits. To isolate this subset, we train a compact binary mask over the edited weights. The mask reverses 80% of edits on the training set and over 70% on the test set, confirming that diverse edits share a common functional structure. Our analysis reveals that the mask reverses edits by eliminating overattention in later layers. Additionally, we show that injecting the mask during editing drops editing success from 98% to 38%, demonstrating that this mechanism is necessary for edits to succeed. Our finding that edits suppress rather than overwrite knowledge explains why ROME and MEMIT fail to propagate changes to related facts. The identified common functional subspace informs detection and defense against unwanted edits 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Locating and Editing Factual Associations in GPT
  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
  • Massive Editing for Large Language Models via Meta Learning cites Locating and Editing Factual Associations in GPT
  • Massive Editing for Large Language Models via Meta Learning cites Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
  • WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models cites Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
  • WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models cites Locating and Editing Factual Associations in GPT
  • Can Fine-Tuning Erase Edits? On the Fragile Coexistence of Knowledge Editing and Fine-tuning cites Locating and Editing Factual Associations in GPT
  • Can Fine-Tuning Erase Edits? On the Fragile Coexistence of Knowledge Editing and Fine-tuning cites WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Can Fine-Tuning Erase Edits? On the Fragile Coexistence of Knowledge Editing and Fine-tuningYinjie Cheng, Paul Youssef, Christin Seifert, Jörg Schlötterer et al.KDD 2026 · 2 citations

Builds on17

  1. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  2. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value AdaptorsTom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim et al.NeurIPS 2023 · 349 citations
  3. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language ModelsPeter Hase, Mohit Bansal, Been Kim, Asma GhandehariounNeurIPS 2023 · 307 citations
  4. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language ModelsPeng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu et al.NeurIPS 2024 · 125 citations
  5. Massive Editing for Large Language Models via Meta LearningChenmien Tan, Ge Zhang, Jie FuICLR 2024 · 68 citations

Related papers

  1. Model Editing at Scale leads to Gradual and Catastrophic ForgettingAkshat Gupta, Anurag Rao, Gopala AnumanchipalliACL Findings 2024 · 12 citations
  2. What does the Knowledge Neuron Thesis Have to do with Knowledge?Jingcheng Niu, Andrew Liu, Zining Zhu, Gerald PennICLR 2024 · 53 citations
  3. A Unified Framework for Model EditingAkshat Gupta, Dev Sajnani, Gopala AnumanchipalliEMNLP Findings 2024 · 10 citations
  4. Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledgeDaniel Mela, Aitor Gonzalez-Agirre, Javier Hernando, Marta VillegasACL Findings 2024 · 3 citations
  5. Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language ModelsXiyu Liu, Zhengxiao Liu, Naibin Gu, Zheng Lin et al.AAAI 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines