Lune

ACL Findings2024

SSS: Editing Factual Knowledge in Language Models towards Semantic Sparse Space

Huazheng Wang, Haifeng Sun, Jingyu Wang, Qi Qi, Zixuan Xia, Menghao Zhang, Jianxin Liao

Abstract

Language Models (LMs) acquire factual knowledge during pre-training and store it in the parameters, which can be valuable for downstream tasks. As world evolves, some facts may be incorrectly induced or become obsolete over time. Various model editing methods have been proposed to modify specific examples in LMs. However, existing training-based methods still suffer from sub-optimal locality, where irrelevant neighborhood examples can be adversely influenced. Model's gradients are still struggling to identify the appropriate direction when updating the parameters. To address this issue, we find that directing the hidden state of the edit example towards spaces where semantics are sparse tends to help preserve the semantics of irrelevant neighborhood examples. Based on this hypothesis, we propose a novel metric, named SSS, to evaluate the degree of sparsity around a sentence embedding in the semantic space without any human or machine annotation. Subsequently, we incorporate SSS into the original loss function of the existing training-based methods to enhance locality. Experiments conducted on two datasets across various models demonstrate that SSS is effective in improving both locality and reasoning capability. Code will be available at: https://github.com/MaybeLizzy/EditSSS.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Editable Neural Networks
  • Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Locating and Editing Factual Associations in GPT
  • Editing Large Language Models: Problems, Methods, and Opportunities cites Editable Neural Networks
  • Editing Large Language Models: Problems, Methods, and Opportunities cites Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
  • Editing Large Language Models: Problems, Methods, and Opportunities cites Locating and Editing Factual Associations in GPT
  • Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models cites Locating and Editing Factual Associations in GPT
  • Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models cites Editing Large Language Models: Problems, Methods, and Opportunities
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language ModelsHuazheng Wang, Yongcheng Jing, Haifeng Sun, Yingjie Wang et al.ACL 2026 · 3 citations

Builds on18

  1. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  2. On the Sentence Embeddings from Pre-trained Language ModelsBohan Li, Hao Zhou, Junxian He, Mingxuan Wang et al.EMNLP 2020 · 538 citations
  3. Aging with GRACE: Lifelong Model Editing with Discrete Key-Value AdaptorsTom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim et al.NeurIPS 2023 · 349 citations
  4. Editable Neural NetworksAnton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov et al.ICLR 2020 · 210 citations
  5. Editing Large Language Models: Problems, Methods, and OpportunitiesYunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng et al.EMNLP 2023 · 83 citations

Related papers

  1. STEAM: A Semantic-Level Knowledge Editing Framework for Large Language ModelsGeunyeong Jeong, Juoh Sun, Seonghee Lee, Harksoo KimEMNLP Findings 2025 · 1 citation
  2. EAMET: Robust Massive Model Editing via Embedding Alignment OptimizationYanbo Dai, Zhenlan Ji, Zongjie Li, Shuai WangICLR 2026
  3. SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding AlteringXiaopeng Li, Shasha Li, Shezheng Song, Huijun Liu et al.AAAI 2025 · 11 citations
  4. Neighboring Perturbations of Knowledge Editing on Large Language ModelsJun-Yu Ma, Zhen-Hua Ling, Ningyu Zhang, Jia-Chen GuICML 2024 · 6 citations
  5. Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model EditingBhiman Kumar Baghel, Emma Jordan, Zheyuan Ryan Shi, Xiang Lorraine LiEMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines