ACL Findings2024
Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models
Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria
Abstract
Warning: This paper contains several unethical and sensitive statements.
In the rapidly advancing field of artificial intelligence, the concept of 'Red-Teaming' or 'Jailbreaking' large language models (LLMs) has emerged as a crucial area of study. This approach is especially significant in terms of assessing and enhancing the safety and robustness of these models. This paper investigates the intricate consequences of such modifications through model editing, uncovering a complex relationship between enhancing model accuracy and preserving its ethical integrity. Our in-depth analysis reveals a striking paradox: while injecting accurate information is crucial for model reliability, it can paradoxically destabilize the model's foundational framework, resulting in unpredictable and potentially unsafe behaviors. Additionally, we propose a benchmark dataset NICHEHAZARDQA to investigate this unsafe behavior both within the same and cross topical domain. This aspect of our research sheds light on how the edits, impact the model's safety metrics and guardrails. Our findings show that model editing serves as a cost-effective tool for topical red-teaming by methodically applying targeted edits and evaluating the resultant model behavior 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Detoxifying Large Language Models via Knowledge Editing cites Locating and Editing Factual Associations in GPT
- Revealing and Mitigating Over-Attention in Knowledge Editing cites Locating and Editing Factual Associations in GPT
- Revealing and Mitigating Over-Attention in Knowledge Editing cites Memory-Based Model Editing at Scale
- Revealing and Mitigating Over-Attention in Knowledge Editing cites Unveiling the Pitfalls of Knowledge Editing for Large Language Models
- Tracing and Reversing Edits in LLMs cites Locating and Editing Factual Associations in GPT
- Tracing and Reversing Edits in LLMs cites Memory-Based Model Editing at Scale
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models cites Locating and Editing Factual Associations in GPT
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models cites Memory-Based Model Editing at Scale
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- Erasing Concepts from Diffusion Models cites Locating and Editing Factual Associations in GPT
Cited by top-tier papers8
- Unveiling the Pitfalls of Knowledge Editing for Large Language ModelsZhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang et al.ICLR 2024 · 47 citations
- Tracing and Reversing Edits in LLMsPaul Youssef, Zhixue Zhao, Christin Seifert, Jörg SchlöttererICLR 2026 · 7 citations
- "Flex Tape Can't Fix That": Bias and Misinformation in Edited Language ModelsKarina Halevy, Anna Sotnikova, Badr AlKhamissi, Syrielle Montariol et al.EMNLP 2024 · 1 citation
- Detoxifying Large Language Models via Knowledge EditingMengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi et al.ACL 2024
- Revealing and Mitigating Over-Attention in Knowledge EditingPinzheng Wang, Zecheng Tang, Keyan Zhou, Juntao Li et al.ICLR 2025
Builds on15
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Erasing Concepts from Diffusion ModelsRohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, David BauICCV 2023 · 536 citations
- Memory-Based Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning et al.ICML 2022 · 520 citations
Related papers
- Jailbreaking Large Language Models Against Moderation Guardrails via Cipher CharactersHaibo Jin, Andy Zhou, Joe D. Menke, Haohan WangNeurIPS 2024 · 55 citations
- SAME: Safety-Aware Model Editing Guided by Safety TransformationJiayi Wang, Shipeng Wang, Ji Wu, Jian SunACL 2026
- A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial RobustnessLeo Schwinn, Moritz Ladenburger, Tim Beyer, Mehrnaz Mofakhami et al.ICML 2026 · 15 citations
- GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak MethodsRuixuan Huang, Xunguang Wang, Zongjie Li, Daoyuan Wu et al.ICLR 2026 · 11 citations
- CoP: Agentic Red-teaming for Large Language Models using Composition of PrinciplesChen Xiong, Pin-Yu Chen, Tsung-Yi HoNeurIPS 2025 · 13 citations
