TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis
Xiaorui Wu, Xiaofeng Mao, Fei Li, Xin Zhang, Xuanhong Li, Chong Teng, Donghong Ji, Zhuang Li
Abstract
Large Language Models (LLMs) excel in various natural language processing tasks but remain vulnerable to generating harmful content or being exploited for malicious purposes. Although safety alignment datasets have been introduced to mitigate such risks through supervised fine-tuning (SFT), these datasets often lack comprehensive risk coverage. Most existing datasets focus primarily on lexical diversity while neglecting other critical dimensions. To address this limitation, we propose a novel analysis framework to systematically measure the risk coverage of alignment datasets across three essential dimensions: Lexical Diversity, Malicious Intent, and Jailbreak Tactics. We further introduce TRIDENT, an automated pipeline that leverages personabased, zero-shot LLM generation to produce diverse and comprehensive instructions spanning these dimensions. Each harmful instruction is paired with an ethically aligned response, resulting in two datasets: TRIDENT-CORE, comprising 26,311 examples, and TRIDENT-EDGE, with 18,773 examples. Fine-tuning META-LLAMA-3.1-8B on TRIDENT-EDGE demonstrates substantial improvements, achieving an average 14.29% reduction in Harm Score, and a 20% decrease in Attack Success Rate compared to the best-performing baseline model fine-tuned on the WILDBREAK dataset. Our datasets are available at https: //github.com/FishT0ucher/TRIDENT . 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Cited by top-tier papers2
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious InstructionsXiaorui Wu, Fei Li, Xiaofeng Mao, Xin Zhang et al.NeurIPS 2025 · 8 citations
- AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMsDebdeep Sanyal, Manodeep Ray, Murari MandalAAAI 2026 · 2 citations
Builds on12
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen et al.ICLR 2024 · 1,104 citations
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
- WildChat: 1M ChatGPT Interaction Logs in the WildWenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie et al.ICLR 2024 · 504 citations
Related papers
- Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak AttacksYingjie Zhang, Tong Liu, Zhe Zhao, Guozhu Meng et al.NDSS 2026 · 5 citations
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language ModelsYongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang et al.ICML 2024 · 140 citations
- Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential AttacksSheng Liu, Qiang Sheng, Danding Wang, Yang Liu et al.EMNLP Findings 2025 · 1 citation
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?Yanbo Wang, Jiyang Guan, Jian Liang, Ran HeCVPR 2025
- Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language ModelsKai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang et al.ACL 2026
