Lune

ICLR2025Top-tier venue

Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models

Guobin Shen, Dongcheng Zhao, Yiting Dong, Xiang He, Yi Zeng

Abstract

As large language models (LLMs) become integral to various applications, ensuring both their safety and utility is paramount. Jailbreak attacks, which manipulate LLMs into generating harmful content, pose significant challenges to this balance. Existing defenses, such as prompt engineering and safety fine-tuning, often introduce computational overhead, increase inference latency, and lack runtime flexibility. Moreover, overly restrictive safety measures can degrade model utility by causing refusals of benign queries. In this paper, we introduce Jailbreak Antidote, a method that enables real-time adjustment of LLM safety preferences by manipulating a sparse subset of the model's internal states during inference. By shifting the model's hidden representations along a safety direction with varying strengths, we achieve flexible control over the safety-utility balance without additional token overhead or inference delays. Our analysis reveals that safety-related information in LLMs is sparsely distributed; adjusting approximately 5% of the internal state is as effective as modifying the entire state. Extensive experiments on nine LLMs (ranging from 2 billion to 72 billion parameters), evaluated against ten jailbreak attack methods and compared with six defense strategies, validate the effectiveness and efficiency of our approach. By directly manipulating internal states during reasoning, Jailbreak Antidote offers a lightweight, scalable solution that enhances LLM safety while preserving utility, opening new possibilities for real-time safety mechanisms in widely-deployed AI systems.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Locating and Editing Factual Associations in GPT cites Language Models are Few-Shot Learners
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models cites Measuring Massive Multitask Language Understanding
  • Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
  • Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
  • Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint cites Language Models are Few-Shot Learners
  • AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. AlphaSteer: Learning Refusal Steering with Principled Null-Space ConstraintLeheng Sheng, Changshuo Shen, Weixiang Zhao, Junfeng Fang et al.ICLR 2026 · 52 citations
  2. Combining Cost Constrained Runtime Monitors for AI SafetyTim Tian Hua, James Baskerville, Henri Lemoine, Mia Hopman et al.NeurIPS 2025 · 12 citations
  3. Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-DefenseGuobin Shen, Dongcheng Zhao, Haibo Tong, Jindong Li et al.ICLR 2026 · 4 citations
  4. SAFENUDGE: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offsJoão Fonseca, Andrew Bell, Julia StoyanovichEMNLP 2025 · 1 citation
  5. Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language ModelsGuangyu Yang, Jinghong Chen, Jingbiao Mei, Weizhe Lin et al.ACL 2026 · 1 citation

Builds on24

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  4. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  5. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations

Related papers

  1. DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model EditingYi Wang, Fenghua Weng, Sibei Yang, Zhan Qin et al.ACL Findings 2025 · 2 citations
  2. SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation InterventionJiaqi Wu, Chen Chen, Chunyan Hou, Xiaojie YuanEMNLP Findings 2025 · 5 citations
  3. DeTAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention ModificationYu Li, Han Jiang, Zhihua WeiACL Findings 2025 · 4 citations
  4. You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak DefenseWuyuao Mai, Geng Hong, Pei Chen, Xudong Pan et al.WWW 2025 · 9 citations
  5. Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMsZhixin Xie, Xurui Song, Jun LuoNeurIPS 2025 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines