ACL Findings2025
RelEdit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning
Yifan Niu, Miao Peng, Nuo Chen, Yatao Bian, Tingyang Xu, Jia Li
Abstract
The conceptual knowledge in Large Language Models (LLMs) can become outdated over time, and concept editing is often an option. Current evaluations on conceptual knowledge editing primarily focus on whether the definitions of concepts are successfully edited, neglecting the impact on the model's related beliefs. To address this gap, we introduce a benchmark called RelEdit, which includes criteria and questions to assess both concept-level and instance-level relational reasoning abilities of edited models. Our findings reveal that existing knowledge editing methods struggle to reason about related conceptual knowledge effectively. Additionally, we introduce a simple memory-based in-context editing baseline, MICE, which prompts the language model to generate answers that align with the stored edited concepts in external memory. In addition, we find that MICE obtains the best scores on our benchmark, suggesting a promising research direction for model editing. The code is available at https://github.com/ivanniu/RelEdit.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Memory-Based Model Editing at Scale cites Editable Neural Networks
- Memory-Based Model Editing at Scale cites Language Models are Few-Shot Learners
- Fast Model Editing at Scale cites Editable Neural Networks
- Fast Model Editing at Scale cites Language Models are Few-Shot Learners
- Editing Large Language Models: Problems, Methods, and Opportunities cites Editable Neural Networks
- Editing Large Language Models: Problems, Methods, and Opportunities cites Language Models are Few-Shot Learners
Builds on17
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
- Memory-Based Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning et al.ICML 2022 · 520 citations
- Editable Neural NetworksAnton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov et al.ICLR 2020 · 210 citations
- Editing Large Language Models: Problems, Methods, and OpportunitiesYunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng et al.EMNLP 2023 · 83 citations
Related papers
- Editing Conceptual Knowledge for Large Language ModelsXiaohan Wang, Shengyu Mao, Shumin Deng, Yunzhi Yao et al.EMNLP Findings 2024 · 9 citations
- Towards Meta-Cognitive Knowledge Editing for Multimodal LLMsZhaoyu Fan, Kaihang Pan, Mingze Zhou, Bosheng Qin et al.WWW 2026
- MultiMedBench: A Scenario-Aware Benchmark for Evaluating Knowledge Editing in Medical VQAShengtao Wen, Haodong Chen, Yadong Wang, Zhongying Pan et al.AAAI 2026
- Uncovering Overfitting in Large Language Model EditingMengqi Zhang, Xiaotian Ye, Qiang Liu, Shu Wu et al.ICLR 2025
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge EditingYaohui Ma, Xiaopeng Hong, Shizhou Zhang, Huiyun Li et al.AAAI 2025 · 2 citations
