Lune

EMNLP Findings2025

ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts

Amelia Hardy, Houjun Liu, Allie Griffith, Bernard Lange, Duncan Eddy, Mykel J. Kochenderfer

Abstract

Existing LLM red-teaming approaches prioritize high attack success rate, often resulting in high-perplexity prompts. This focus overlooks low-perplexity attacks that are more difficult to filter, more likely to arise during benign usage, and more impactful as negative downstream training examples. In response, we introduce ASTPrompter, a single-step optimization method that uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate (ASR). ASTPrompter achieves an attack success rate 5.1 times higher on Llama-8.1B while using inputs that are 2.1 times more likely to occur according to the frozen LLM. Furthermore, our attack transfers to Mistral-7B, Qwen-7B, and TinyLlama in both black-and white-box settings. Lastly, by tuning a single hyperparameter in our method, we discover successful attack prefixes along an efficient frontier between ASR and perplexity, highlighting perplexity as a previously under-considered factor in red-teaming.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models cites The Curious Case of Neural Text Degeneration
  • Jailbroken: How Does LLM Safety Training Fail? cites RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on18

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  3. Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob SteinhardtNeurIPS 2023 · 2,230 citations
  4. Tree of Attacks: Jailbreaking Black-Box LLMs AutomaticallyAnay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson et al.NeurIPS 2024 · 835 citations
  5. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi et al.EMNLP Findings 2020 · 692 citations

Related papers

  1. AdvPrompter: Fast Adaptive Adversarial Prompting for LLMsAnselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos et al.ICML 2025
  2. Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring TechniquePala Tej Deep, Vernon Toh, Rishabh Bhardwaj, Soujanya PoriaEMNLP Findings 2025
  3. ProAdvPrompter: A Two-Stage Journey to Effective Adversarial Prompting for LLMsHao Di, Tong He, Haishan Ye, Yinghui Huang et al.ICLR 2025
  4. STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red TeamingMinJae Jung, YongTaek Lim, Chaeyun Kim, Junghwan Kim et al.ACL Findings 2026
  5. Active Attacks: Red-teaming LLMs via Adaptive EnvironmentsTaeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio et al.ICML 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines