Lune

NeurIPS2024Top-tier venue

Representation Noising: A Defence Mechanism Against Harmful Finetuning

Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Carsten Maple, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

41 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Erasing Conceptual Knowledge from Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
  • Obfuscated Activations Bypass LLM Latent-Space Defenses cites Erasing Conceptual Knowledge from Language Models
  • Obfuscated Activations Bypass LLM Latent-Space Defenses cites Refusal in Language Models Is Mediated by a Single Direction
  • Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs cites Measuring Massive Multitask Language Understanding
  • Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs cites Erasing Conceptual Knowledge from Language Models
  • AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents cites Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
  • Refusal in Language Models Is Mediated by a Single Direction cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers41

  1. Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt TemplatesKaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu et al.NeurIPS 2024 · 131 citations
  2. AgentAuditor: Human-level Safety and Security Evaluation for LLM AgentsHanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li et al.NeurIPS 2025 · 98 citations
  3. Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMsKyle O'Brien, Stephen Casper, Quentin Anthony, Tomek Korbak et al.ICLR 2026 · 59 citations
  4. Erasing Conceptual Knowledge from Language ModelsRohit Gandikota, Sheridan Feucht, Samuel Marks, David BauNeurIPS 2025 · 35 citations
  5. Obfuscated Activations Bypass LLM Latent-Space DefensesLuke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov et al.ICLR 2026 · 28 citations

Builds on24

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. Refusal in Language Models Is Mediated by a Single DirectionAndy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka et al.NeurIPS 2024 · 1,166 citations

Related papers

  1. Representation Bending for Large Language Model SafetyAshkan Yousefpour, Taeheon Kim, Ryan Sungmo Kwon, Seungbeen Lee et al.ACL 2025 · 19 citations
  2. One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMsYixin Tan, Yu Zhe, Rui Wen, Jun SakumaCCS 2026
  3. SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation InterventionJiaqi Wu, Chen Chen, Chunyan Hou, Xiaojie YuanEMNLP Findings 2025 · 5 citations
  4. Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding SpaceLeo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel et al.NeurIPS 2024 · 113 citations
  5. Tamper-Resistant Safeguards for Open-Weight LLMsRishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines