EMNLP Findings2024
DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLMs Jailbreakers
Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, Cho-Jui Hsieh
Abstract
Safety-aligned Large Language Models (LLMs) are still vulnerable to some manual and automated jailbreak attacks, which adversarially trigger LLMs to output harmful content.
However, existing jailbreaking methods usually view a harmful prompt as a whole but they are not effective at reducing LLMs' attention on combinations of words with malice, which well-aligned LLMs can easily reject. This paper discovers that decomposing a malicious prompt into separated sub-prompts can effectively reduce LLMs' attention on harmful words by presenting them to LLMs in a fragmented form, thereby addressing these limitations and improving attack effectiveness. We introduce an automatic prompt Decomposition and Reconstruction framework for jailbreaking Attack (DrAttack). DrAttack consists of three key components: (a) 'Decomposition' of the original prompt into sub-prompts, (b) 'Reconstruction' of these sub-prompts implicitly by In-Context Learning with semantically similar but benign reassembling example, and (c) 'Synonym Search' of sub-prompts, aiming to find sub-prompts' synonyms that maintain the original intent while jailbreaking LLMs. An extensive empirical study across multiple open-source and closed-source LLMs demonstrates that, with fewer queries, DrAttack obtains a substantial gain of success rate on powerful LLMs over prior SOTA attackers. Notably, the success rate of 80% on GPT-4 surpassed previous art by 65%.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Sok: Evaluating Jailbreak Guardrails for Large Language Models cites Jailbroken: How Does LLM Safety Training Fail?
- Sok: Evaluating Jailbreak Guardrails for Large Language Models cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output cites Jailbroken: How Does LLM Safety Training Fail?
- When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attacks cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attacks cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attacks cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Cited by top-tier papers12
- Sok: Evaluating Jailbreak Guardrails for Large Language ModelsXunguang Wang, Zhenlan Ji, Wenxuan Wang, Zongjie Li et al.S&P 2026 · 27 citations
- HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL ConferencesYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL 2026 · 18 citations
- GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak MethodsRuixuan Huang, Xunguang Wang, Zongjie Li, Daoyuan Wu et al.ICLR 2026 · 11 citations
- When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured OutputShuoming Zhang, Jiacheng Zhao, Hanyuan Dong, Ruiyuan Xu et al.CCS 2026 · 6 citations
- Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak AttacksYingjie Zhang, Tong Liu, Zhe Zhao, Guozhu Meng et al.NDSS 2026 · 5 citations
Builds on20
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob SteinhardtNeurIPS 2023 · 2,230 citations
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
Related papers
- Exploiting Task-Level Vulnerabilities: An Automatic Jailbreak Attack and Defense Benchmarking for LLMsLan Zhang, Xinben Gao, Liuyi Yao, Jinke Song et al.USENIX Security 2025
- Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and ReconstructionTong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong et al.USENIX Security 2024 · 121 citations
- Dual Intention Escape: Penetrating and Toxic Jailbreak Attack against Large Language ModelsYanni Xue, Jiakai Wang, Zixin Yin, Yuqing Ma et al.WWW 2025 · 5 citations
- Stand on The Shoulders of Giants: Building JailExpert from Previous Attack ExperienceXi Wang, Songlei Jian, Shasha Li, Xiaopeng Li et al.EMNLP 2025 · 1 citation
- Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMsHaoming Yang, Ke Ma, Xiaojun Jia, Yingfei Sun et al.ICML 2025
