EMNLP Findings2024
Immunization against harmful fine-tuning attacks
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Hassan Sajjad, Frank Rudzicz
Abstract
Large Language Models (LLMs) are often trained with safety guards intended to prevent harmful text generation. However, such safety training can be removed by fine-tuning the LLM on harmful datasets. While this emerging threat (harmful fine-tuning attacks) has been characterized by previous work, there is little understanding of how we should proceed in constructing and validating defenses against these attacks especially in the case where defenders would not have control of the finetuning process. We introduce a formal framework based on the training budget of an attacker which we call "Immunization" conditions. Using a formal characterisation of the harmful fine-tuning problem, we provide a thorough description of what a successful defense must comprise of and establish a set of guidelines on how rigorous defense research that gives us confidence should proceed.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
17 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack cites Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler cites LoRA: Low-Rank Adaptation of Large Language Models
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler cites Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler cites Representation Noising: A Defence Mechanism Against Harmful Finetuning
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler cites Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
- Representation Noising: A Defence Mechanism Against Harmful Finetuning cites LoRA: Low-Rank Adaptation of Large Language Models
- Representation Noising: A Defence Mechanism Against Harmful Finetuning cites Measuring Massive Multitask Language Understanding
- Representation Noising: A Defence Mechanism Against Harmful Finetuning cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Representation Noising: A Defence Mechanism Against Harmful Finetuning cites Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates cites Representation Noising: A Defence Mechanism Against Harmful Finetuning
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates cites Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
Cited by top-tier papers17
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt TemplatesKaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu et al.NeurIPS 2024 · 131 citations
- Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning AttackTiansheng Huang, Sihao Hu, Fatih Ilhan, Selim F. Tekin et al.NeurIPS 2024 · 113 citations
- Representation Noising: A Defence Mechanism Against Harmful FinetuningDomenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze et al.NeurIPS 2024 · 107 citations
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning PerturbationYibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao et al.NeurIPS 2025 · 22 citations
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data SchedulerZixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei et al.NeurIPS 2025 · 16 citations
Builds on21
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen et al.ICLR 2024 · 1,104 citations
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
Related papers
- Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient InfluenceQuoc Minh Nguyen, Trung Le, Jing Wu, Anh Tuan Bui et al.ICLR 2026 · 10 citations
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning AttackTiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Joshua Kimball et al.ICML 2025
- SDD: Self-Degraded Defense against Malicious Fine-tuningZixuan Chen, Weikai Lu, Xin Lin, Ziqian ZengACL 2025
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language ModelsYongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang et al.ICML 2024 · 140 citations
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?Yanbo Wang, Jiyang Guan, Jian Liang, Ran HeCVPR 2025
