Lune

NeurIPS2024Top-tier venue

Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization

Kai Hu, Weichen Yu, Yining Li, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu, Zhiqiang Shen, Kai Chen, Matt Fredrikson

Abstract

Recent research indicates that large language models (LLMs) are susceptible to jailbreaking attacks that can generate harmful content. This paper introduces a novel token-level attack method, Adaptive Dense-to-Sparse Constrained Optimization (ADC), which has been shown to successfully jailbreak multiple open-source LLMs. Drawing inspiration from the difficulties of discrete token optimization, our method relaxes the discrete jailbreak optimization into a continuous optimization process while gradually increasing the sparsity of the optimizing vectors. This technique effectively bridges the gap between discrete and continuous space optimization. Experimental results demonstrate that our method is more effective and efficient than state-of-the-art token-level methods. On Harmbench, our approach achieves the highest attack success rate on seven out of eight LLMs compared to the latest jailbreak methods. Trigger Warning: This paper contains model behavior that can be offensive in nature.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Automatic Dialectic Jailbreak: A Framework for Generating Effective Jailbreak Strategies cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • Automatic Dialectic Jailbreak: A Framework for Generating Effective Jailbreak Strategies cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment Training cites Training language models to follow instructions with human feedback
  • Tree of Attacks: Jailbreaking Black-Box LLMs Automatically cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion cites Training language models to follow instructions with human feedback
  • Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • Training language models to follow instructions with human feedback cites RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
  • AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired SearchXun Huang, Simeng Qin, Xiaoshuang Jia, Ranjie Duan et al.ICLR 2026 · 9 citations
  2. Jailbreaking the Matrix: Nullspace Steering for Controlled Model SubversionVishal Pramanik, Maisha Maliha, Susmit Jha, Sumit Kumar JhaICLR 2026 · 3 citations
  3. LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment TrainingYuyang Gong, Zihao Wang, Jiawei Liu, XiaoFeng WangCCS 2026 · 1 citation
  4. Automatic Dialectic Jailbreak: A Framework for Generating Effective Jailbreak StrategiesJianghai Yu, Yang Zhou, Zihan Zhou, Lingjuan Lyu et al.ICLR 2026
  5. Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language ModelsKai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang et al.ACL 2026

Builds on22

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  3. Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
  4. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
  5. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi et al.EMNLP Findings 2020 · 692 citations

Related papers

  1. TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language ModelsZhi Xu, Jiaqi Li, Xiaotong Zhang, Hong Yu et al.ICLR 2026 · 2 citations
  2. Safety Guardrails of Large Language Models Are Vulnerable to Value-Driven Adversarial PromptingXiaohao Luo, Ying Wei, Zhijun LiACL Findings 2026
  3. Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous ConstraintsJunxiao Yang, Zhexin Zhang, Shiyao Cui, Hongning Wang et al.ACL 2025 · 6 citations
  4. Exploring Jailbreak Attacks on LLMs through Intent Concealment and DiversionTiehan Cui, Yanxu Mao, Peipei Liu, Congying Liu et al.ACL Findings 2025 · 3 citations
  5. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive AttacksMaksym Andriushchenko, Francesco Croce, Nicolas FlammarionICLR 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines