Lune

ICML2026Top-tier venue

Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework

David Huang, Jaewon Chang, Avidan Shah, Prateek Mittal, Chawin Sitawarin

Abstract

The Rapid Response (RR) framework (Peng et al., 2024) , deployed in production systems, including Anthropic's ASL-3 safeguards (Anthropic, 2025), continuously improves jailbreak-detection classifiers. When new jailbreaks emerge that bypass these classifiers, Rapid Response generates synthetic variants for training, helping the model generalize from the new attacks and quickly adapt. We reveal that prompt injection can infiltrate this pipeline to deliver poisoned samples into the classifier's training set, enabling two attack objectives: (I) targeted poisoning attacks that create false positives on harmless samples by categorizing them as a jailbreak, with a specific desired feature (e.g., certain formatting, subject, or keyword), (II) concept-based backdoor attacks that induce false negatives on jailbreak inputs, generalizing even to jailbreaks from attack strategies the defender explicitly trained against, when the backdoor trigger is present. Importantly, our threat model restricts adversaries to modifying only jailbreak samples (not benign data or labels), a constraint unexplored by prior work that makes the second objective particularly challenging. We address this with Omission Attack, which exploits a new phenomenon: when training on concept-absent unsafe samples, the classifier misassociates that concept's presence with the safe label. Both attacks cause substantial and in some cases near-complete label flipping at only a 1% poisoning rate, achieving up to 100% false positive rates and up to 96% false negative rates. Code: https://github.com/ DH-davidhuang/rapid-poison .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Refusal in Language Models Is Mediated by a Single Direction cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on19

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
  3. Refusal in Language Models Is Mediated by a Single DirectionAndy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka et al.NeurIPS 2024 · 1,166 citations
  4. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen et al.ICLR 2024 · 1,104 citations
  5. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations

Related papers

  1. Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable RewardWeiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou et al.ACL 2026 · 4 citations
  2. Universal Jailbreak Backdoors from Poisoned Human FeedbackJavier Rando, Florian TramèrICLR 2024 · 124 citations
  3. Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language ModelsGuangyu Yang, Jinghong Chen, Jingbiao Mei, Weizhe Lin et al.ACL 2026 · 1 citation
  4. From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMsGuangyu Shen, Siyuan Cheng, Xiangzhe Xu, Yuan Zhou et al.ICML 2026
  5. Steering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation SimulationLUOYU CHEN, Weiqi Wang, Zhiyi Tian, Chenhan Zhang et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines