EMNLP Findings2025
ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts
Amelia Hardy, Houjun Liu, Allie Griffith, Bernard Lange, Duncan Eddy, Mykel J. Kochenderfer
Abstract
Existing LLM red-teaming approaches prioritize high attack success rate, often resulting in high-perplexity prompts. This focus overlooks low-perplexity attacks that are more difficult to filter, more likely to arise during benign usage, and more impactful as negative downstream training examples. In response, we introduce ASTPrompter, a single-step optimization method that uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate (ASR). ASTPrompter achieves an attack success rate 5.1 times higher on Llama-8.1B while using inputs that are 2.1 times more likely to occur according to the frozen LLM. Furthermore, our attack transfers to Mistral-7B, Qwen-7B, and TinyLlama in both black-and white-box settings. Lastly, by tuning a single hyperparameter in our method, we discover successful attack prefixes along an efficient frontier between ASR and perplexity, highlighting perplexity as a previously under-considered factor in red-teaming.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models cites The Curious Case of Neural Text Degeneration
- Jailbroken: How Does LLM Safety Training Fail? cites RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Builds on18
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob SteinhardtNeurIPS 2023 · 2,230 citations
- Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi et al.EMNLP Findings 2020 · 692 citations
Related papers
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMsAnselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos et al.ICML 2025
- Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring TechniquePala Tej Deep, Vernon Toh, Rishabh Bhardwaj, Soujanya PoriaEMNLP Findings 2025
- ProAdvPrompter: A Two-Stage Journey to Effective Adversarial Prompting for LLMsHao Di, Tong He, Haishan Ye, Yinghui Huang et al.ICLR 2025
- STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red TeamingMinJae Jung, YongTaek Lim, Chaeyun Kim, Junghwan Kim et al.ACL Findings 2026
- Active Attacks: Red-teaming LLMs via Adaptive EnvironmentsTaeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio et al.ICML 2026 · 5 citations
