Lune

EMNLP Findings2025

Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique

Pala Tej Deep, Vernon Toh, Rishabh Bhardwaj, Soujanya Poria

Abstract

As large language models (LLMs) are increasingly integrated into real-world applications, ensuring their safety and robustness is critical. Automated red-teaming methods generate adversarial attacks to identify vulnerabilities, but existing approaches often face challenges like slow performance, limited categorical diversity, and high resource demands. We propose FERRET, a novel method that enhances the baseline, RAINBOW TEAMING by generating multiple adversarial prompt mutations per iteration and ranking them using scoring functions such as reward models, Llama Guard, and LLM-as-a-judge. FERRET achieves a 95% attack success rate (ASR), a 46% improvement over baseline, and reduces time to a 90% ASR by 15.2%. Additionally, it generates transferable adversarial prompts effective on larger LLMs. Our code is available at https: //github.com/declare-lab/ferret

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites Red Teaming Language Models with Language Models
  • The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • Tree of Attacks: Jailbreaking Black-Box LLMs Automatically cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints cites Red Teaming Language Models with Language Models
  • DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints cites Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
  • Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
  • Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites Red Teaming Language Models with Language Models
  • Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites Red Teaming Language Models with Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing ConstraintsAndrew Zhao, Quentin Xu, Matthieu Lin, Shenzhi Wang et al.AAAI 2025 · 11 citations
  2. The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree SearchRongzhe Wei, Peizhi Niu, Xinjie Shen, Tony Tu et al.ICML 2026 · 5 citations

Builds on4

  1. Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
  2. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
  3. Red Teaming Language Models with Language ModelsEthan Perez, Saffron Huang, H. Francis Song, Trevor Cai et al.EMNLP 2022 · 239 citations
  4. Rainbow Teaming: Open-Ended Generation of Diverse Adversarial PromptsMikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro et al.NeurIPS 2024 · 231 citations

Related papers

  1. AdvPrompter: Fast Adaptive Adversarial Prompting for LLMsAnselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos et al.ICML 2025
  2. Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety TuningSeanie Lee, Minsu Kim, Lynn Cherif, David Dobre et al.ICLR 2025
  3. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
  4. Attack Prompt Generation for Red Teaming and Defending Large Language ModelsBoyi Deng, Wenjie Wang, Fuli Feng, Yang Deng et al.EMNLP Findings 2023 · 35 citations
  5. Active Attacks: Red-teaming LLMs via Adaptive EnvironmentsTaeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio et al.ICML 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines