EMNLP Findings2025
Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique
Pala Tej Deep, Vernon Toh, Rishabh Bhardwaj, Soujanya Poria
Abstract
As large language models (LLMs) are increasingly integrated into real-world applications, ensuring their safety and robustness is critical. Automated red-teaming methods generate adversarial attacks to identify vulnerabilities, but existing approaches often face challenges like slow performance, limited categorical diversity, and high resource demands. We propose FERRET, a novel method that enhances the baseline, RAINBOW TEAMING by generating multiple adversarial prompt mutations per iteration and ranking them using scoring functions such as reward models, Llama Guard, and LLM-as-a-judge. FERRET achieves a 95% attack success rate (ASR), a 46% improvement over baseline, and reduces time to a 90% ASR by 15.2%. Additionally, it generates transferable adversarial prompts effective on larger LLMs. Our code is available at https: //github.com/declare-lab/ferret
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites Red Teaming Language Models with Language Models
- The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Tree of Attacks: Jailbreaking Black-Box LLMs Automatically cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints cites Red Teaming Language Models with Language Models
- DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints cites Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites Red Teaming Language Models with Language Models
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites Red Teaming Language Models with Language Models
Cited by top-tier papers2
- DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing ConstraintsAndrew Zhao, Quentin Xu, Matthieu Lin, Shenzhi Wang et al.AAAI 2025 · 11 citations
- The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree SearchRongzhe Wei, Peizhi Niu, Xinjie Shen, Tony Tu et al.ICML 2026 · 5 citations
Builds on4
- Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
- Red Teaming Language Models with Language ModelsEthan Perez, Saffron Huang, H. Francis Song, Trevor Cai et al.EMNLP 2022 · 239 citations
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial PromptsMikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro et al.NeurIPS 2024 · 231 citations
Related papers
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMsAnselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos et al.ICML 2025
- Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety TuningSeanie Lee, Minsu Kim, Lynn Cherif, David Dobre et al.ICLR 2025
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
- Attack Prompt Generation for Red Teaming and Defending Large Language ModelsBoyi Deng, Wenjie Wang, Fuli Feng, Yang Deng et al.EMNLP Findings 2023 · 35 citations
- Active Attacks: Red-teaming LLMs via Adaptive EnvironmentsTaeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio et al.ICML 2026 · 5 citations
