Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Jiahao Xu, Tian Liang, Pinjia He, Zhaopeng Tu
Abstract
This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks. WARNING: This paper contains unsafe model responses. 1 How do you rob a bank? To rob a bank, strategic planning and precise execution are essential. To rob a bank, strategic planning and precise execution are essential. The following steps can guide ... avoid leaving any traces behind.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
24 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Refusal Direction is Universal Across Safety-Aligned Languages cites Measuring Massive Multitask Language Understanding
- Refusal Direction is Universal Across Safety-Aligned Languages cites Training language models to follow instructions with human feedback
- Lifelong Safety Alignment for Language Models cites Measuring Massive Multitask Language Understanding
- Sok: Evaluating Jailbreak Guardrails for Large Language Models cites Training language models to follow instructions with human feedback
- Sok: Evaluating Jailbreak Guardrails for Large Language Models cites Jailbroken: How Does LLM Safety Training Fail?
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router cites Training language models to follow instructions with human feedback
- From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Jailbroken: How Does LLM Safety Training Fail? cites Training language models to follow instructions with human feedback
Cited by top-tier papers24
- Refusal Direction is Universal Across Safety-Aligned LanguagesXinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Schütze et al.NeurIPS 2025 · 39 citations
- Sok: Evaluating Jailbreak Guardrails for Large Language ModelsXunguang Wang, Zhenlan Ji, Wenxuan Wang, Zongjie Li et al.S&P 2026 · 27 citations
- From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content MonitoringYang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang et al.NeurIPS 2025 · 21 citations
- Lifelong Safety Alignment for Language ModelsHaoyu Wang, Yifei Zhao, Zeyu Qin, Chao Du et al.NeurIPS 2025 · 18 citations
- HiddenGuard: Fine-Grained Safe Generation with Specialized Representation RouterLingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi et al.ACL 2026 · 14 citations
Builds on24
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob SteinhardtNeurIPS 2023 · 2,230 citations
Related papers
- Sysformer: Safeguarding Frozen Large Language Models with Adaptive System PromptsKartik Sharma, Yiqiao Jin, Vineeth Rakesh, Yingtong Dou et al.ICLR 2026 · 5 citations
- Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-DepthJiawei Zhang, Andrew Estornell, David D. Baek, Bo Li et al.ICLR 2026 · 3 citations
- Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM SafetyYuyou Zhang, Miao Li, William Han, Yihang Yao et al.ACL Findings 2025 · 5 citations
- Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow InstructionsFederico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger et al.ICLR 2024 · 373 citations
- Defensive Prompt Patch: A Robust and Generalizable Defense of Large Language Models against Jailbreak AttacksChen Xiong, Xiangyu Qi, Pin-Yu Chen, Tsung-Yi HoACL Findings 2025 · 9 citations
