HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, Dan Hendrycks
Abstract
Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a largescale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github. com/centerforaisafety/HarmBench .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
290 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Alignment and Robustness with Circuit Breakers cites Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
- Improving Alignment and Robustness with Circuit Breakers cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- Improving Alignment and Robustness with Circuit Breakers cites Image Hijacks: Adversarial Images can Control Generative Models at Runtime
- Improving Alignment and Robustness with Circuit Breakers cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Improving Alignment and Robustness with Circuit Breakers cites Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks
- Tree of Attacks: Jailbreaking Black-Box LLMs Automatically cites Gradient-based Adversarial Attacks against Text Transformers
- Tree of Attacks: Jailbreaking Black-Box LLMs Automatically cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks cites Towards Evaluating the Robustness of Neural Networks
- Gradient-based Adversarial Attacks against Text Transformers cites Towards Evaluating the Robustness of Neural Networks
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models cites Improving Alignment and Robustness with Circuit Breakers
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models cites Gradient-based Adversarial Attacks against Text Transformers
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
Cited by top-tier papers290
- Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
- Improving Alignment and Robustness with Circuit BreakersAndy Zou, Long Phan, Justin Wang, Derek Duenas et al.NeurIPS 2024 · 362 citations
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language ModelsLiwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger et al.NeurIPS 2024 · 247 citations
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking AttacksAndy Zhou, Bo Li, Haohan WangNeurIPS 2024 · 198 citations
Builds on6
- Towards Evaluating the Robustness of Neural NetworksNicholas Carlini, David A. WagnerS&P 2017 · 9,786 citations
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacksFrancesco Croce, Matthias HeinICML 2020 · 2,337 citations
- Image Hijacks: Adversarial Images can Control Generative Models at RuntimeLuke Bailey, Euan Ong, Stuart Russell, Scott EmmonsICML 2024 · 171 citations
- Gradient-based Adversarial Attacks against Text TransformersChuan Guo, Alexandre Sablayrolles, Hervé Jégou, Douwe KielaEMNLP 2021 · 97 citations
- Enhancing Chat Language Models by Scaling High-quality Instructional ConversationsNing Ding, Yulin Chen, Bokai Xu, Yujia Qin et al.EMNLP 2023 · 95 citations
Related papers
- Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to EvaluationAdriana Alvarado Garcia, Ruyuan Wan, Ozioma Collins Oguine, Karla Badillo-UrquiolaCHI 2026 · 1 citation
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMsAnselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos et al.ICML 2025
- A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial RobustnessLeo Schwinn, Moritz Ladenburger, Tim Beyer, Mehrnaz Mofakhami et al.ICML 2026 · 15 citations
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack IntegrationAndy Zhou, Kevin Wu, Francesco Pinto, Zhaorun Chen et al.NeurIPS 2025 · 46 citations
- Attack Prompt Generation for Red Teaming and Defending Large Language ModelsBoyi Deng, Wenjie Wang, Fuli Feng, Yang Deng et al.EMNLP Findings 2023 · 35 citations
