EMNLP Findings2025
UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets
Wenyu Wang, Mengqi Zhang, Xiaotian Ye, Zhaochun Ren, Pengjie Ren, Zhumin Chen
Abstract
Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of GA-based method and its variants on the TOFU and WMDP benchmarks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning-Time Encoding Shapes Unlearning in LLMs cites Measuring Massive Multitask Language Understanding
- Learning-Time Encoding Shapes Unlearning in LLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Learning-Time Encoding Shapes Unlearning in LLMs cites Machine Unlearning
- Certified Data Removal from Machine Learning Models cites Machine Unlearning
- Machine Unlearning cites Certified Data Removal from Machine Learning Models
- LLM Unlearning Should Be Form-Independent cites Measuring Massive Multitask Language Understanding
- LLM Unlearning Should Be Form-Independent cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- LLM Unlearning Should Be Form-Independent cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
Cited by top-tier papers3
- LLM Unlearning Should Be Form-IndependentXiaotian Ye, Mengqi Zhang, Shu WuS&P 2026 · 3 citations
- Learning-Time Encoding Shapes Unlearning in LLMsRuihan Wu, Konstantin Garov, Kamalika ChaudhuriICLR 2026 · 2 citations
- Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language ModelsFuyao Zhang, Xinyu Yan, Tiantong Wu, Wenjie Li et al.AAAI 2026
Builds on30
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Machine UnlearningLucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia et al.S&P 2021 · 1,381 citations
- Certified Data Removal from Machine Learning ModelsChuan Guo, Tom Goldstein, Awni Y. Hannun, Laurens van der MaatenICML 2020 · 633 citations
Related papers
- Explainable LLM Unlearning through ReasoningJunfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu et al.ICLR 2026 · 8 citations
- Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit DifferenceJiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu et al.NeurIPS 2024 · 106 citations
- LLM-Eraser: Optimizing Large Language Model Unlearning through Selective PruningShengming Zhang, Le Zhang, Jingbo Zhou, Zhi Zheng et al.KDD 2025 · 2 citations
- Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMsSungmin Cha, Sungjun Cho, Dasol Hwang, Moontae LeeICLR 2025
- Large Scale Knowledge WashingYu Wang, Ruihan Wu, Zexue He, Xiusi Chen et al.ICLR 2025
