TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards
Xiqiao Xiong, Ouxiang Li, Zhuo Liu, Moxin Li, Wentao Shi, Fengbin Zhu, Qifan Wang, Fuli Feng
Abstract
Large language models have seen widespread adoption, yet they remain vulnerable to multiturn jailbreak attacks, threatening their safe deployment. This has led to the task of training automated multi-turn attackers to probe model safety vulnerabilities. However, existing approaches typically rely on turn-level optimization, which is insufficient for learning long-term attack strategies. To bridge this gap, we formulate this task as a multi-turn reinforcement learning problem, directly optimizing the harmfulness of the final-turn response as the outcome reward. To address the sparse supervision of the outcome reward, we introduce TROJail, which employs two process rewards to evaluate the utility of intermediate prompts and integrate them into advantage estimation. These rewards (1) penalize overly harmful prompts that trigger the model's refusal mechanism, and (2) encourage steering the semantic relevance of responses toward the targeted harmful content. Experimental results show improved attack success rates across multiple models and benchmarks, highlighting the effectiveness of our approach. The code is available at https://github.com/xxiqiao/ TROJail . Warning: This paper contains examples of harmful content.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring cites MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts cites Red Teaming Language Models with Language Models
Cited by top-tier papers2
- Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive ScoringPeichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu et al.ACL 2026 · 8 citations
- Internalizing Safety Understanding in Large Reasoning Models via VerificationYi Zhang, Yuxin Chen, Leheng Sheng, Dongcheng Zhang et al.ICML 2026 · 2 citations
Builds on28
- MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained TransformersWenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong et al.ACL Findings 2021 · 260 citations
- Red Teaming Language Models with Language ModelsEthan Perez, Saffron Huang, H. Francis Song, Trevor Cai et al.EMNLP 2022 · 239 citations
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial PromptsMikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro et al.NeurIPS 2024 · 231 citations
Related papers
- SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak AttacksMingqian Feng, Xiaodong Liu, Weiwei Yang, Jialin Song et al.ICLR 2026 · 13 citations
- MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingWeiyang Guo, Jing Li, Wenya Wang, Yu Li et al.ACL 2025
- TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language ModelsZhi Xu, Jiaqi Li, Xiaotong Zhang, Hong Yu et al.ICLR 2026 · 2 citations
- Multi-Turn Jailbreaking Large Language Models via Attention ShiftingXiaohu Du, Fan Mo, Ming Wen, Tu Gu et al.AAAI 2025 · 26 citations
- Analogy-based Multi-Turn Jailbreak against Large Language ModelsMengjie Wu, Yihao Huang, Zhenjun Lin, Kangjie Chen et al.NeurIPS 2025 · 9 citations
