Lune

ACL Findings2026

LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples

Yezi Liu, Hanning Chen, Wenjun Huang, Yang Ni, Mohsen Imani

Abstract

Large language models (LLMs) possess vast knowledge acquired from extensive training corpora, but they often cannot remove specific pieces of information when needed, which makes it hard to handle privacy, bias mitigation, and knowledge correction. Traditional model unlearning approaches require computationally expensive fine-tuning or direct weight editing, making them impractical for real-world deployment. In this work, we introduce LoRA-based Unlearning with Negative Examples (LUNE 🌙 ), a lightweight framework that performs negative-only unlearning by updating only lowrank adapters while freezing the backbone, thereby localizing edits and avoiding disruptive global changes. Leveraging Low-Rank Adaptation (LoRA), LUNE targets intermediate representations to suppress (or replace) requested knowledge with an order-of-magnitude lower compute and memory than full fine-tuning or direct weight editing. Extensive experiments on multiple factual unlearning tasks show that LUNE: (I) achieves effectiveness comparable to full fine-tuning and memory-editing methods, and (II) reduces computational cost by about an order of magnitude.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning cites Large Language Model Unlearning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Fairness via Independence: A General Regularization Framework for Machine LearningYezi Liu, Hanning Chen, Wenjun Huang, Yang Ni et al.ICLR 2026

Builds on20

  1. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  2. DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
  3. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine TranslationHaoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan et al.ICML 2024 · 447 citations
  4. Large Language Model UnlearningYuanshun Yao, Xiaojun Xu, Yang LiuNeurIPS 2024 · 365 citations
  5. Simplicity Prevails: Rethinking Negative Preference Optimization for LLM UnlearningChongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia et al.NeurIPS 2025 · 182 citations

Related papers

  1. ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language ModelsYujie Lin, Chengyi Yang, Zhishang Xiang, YIPING SONG et al.ICML 2026
  2. ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMsXunlei Chen, Jinyu Guo, Yuang Li, Zhaokun Wang et al.AAAI 2026 · 2 citations
  3. Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMsSungmin Cha, Sungjun Cho, Dasol Hwang, Moontae LeeICLR 2025
  4. Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained KnowledgePengwei Tang, Xiaolin Hu, Yong Liu, Lizhong Ding et al.AAAI 2026 · 6 citations
  5. LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language ModelsJian Liang, Wenke Huang, Guancheng Wan, Qu Yang et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines