Lune

ICML2025Top-tier venue

Safety Alignment Can Be Not Superficial With Explicit Safety Signals

Jianwei Li, Jung-Eun Kim

Abstract

Recent studies on the safety alignment of large language models (LLMs) have revealed that existing approaches often operate superficially, leaving models vulnerable to various adversarial attacks. Despite their significance, these studies generally fail to offer actionable solutions beyond data augmentation for achieving more robust safety mechanisms. This paper identifies a fundamental cause of this superficiality: existing alignment approaches often presume that models can implicitly learn a safety-related reasoning task during the alignment process, enabling them to refuse harmful requests. However, the learned safety signals are often diluted by other competing objectives, leading models to struggle with drawing a firm safety-conscious decision boundary when confronted with adversarial attacks. Based on this observation, by explicitly introducing a safety-related binary classification task and integrating its signals with our attention and decoding strategies, we eliminate this ambiguity and allow models to respond more responsibly to malicious queries. We emphasize that, with less than 0.2x overhead cost, our approach enables LLMs to assess the safety of both the query and the previously generated tokens at each necessary generating step. Extensive experiments demonstrate that our method significantly improves the resilience of LLMs against various adversarial attacks, offering a promising pathway toward more robust generative AI systems.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Measuring Massive Multitask Language Understanding
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Superficial Safety Alignment Hypothesis
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Finetuned Language Models are Zero-Shot Learners
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment cites Training language models to follow instructions with human feedback
  • Safety Alignment of Large Language Models via Contrasting Safe and Harmful Distributions cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Reference cites Measuring Massive Multitask Language Understanding
  • Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Reference cites Superficial Safety Alignment Hypothesis
  • Superficial Safety Alignment Hypothesis cites Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Reference
  • Superficial Safety Alignment Hypothesis cites Measuring Massive Multitask Language Understanding
  • Superficial Safety Alignment Hypothesis cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Superficial Safety Alignment Hypothesis cites Finetuned Language Models are Zero-Shot Learners
  • Superficial Safety Alignment Hypothesis cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Superficial Safety Alignment Hypothesis cites Training language models to follow instructions with human feedback
  • New Wide-Net-Casting Jailbreak Attacks Risk Large Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • New Wide-Net-Casting Jailbreak Attacks Risk Large Models cites Training language models to follow instructions with human feedback
  • Finetuned Language Models are Zero-Shot Learners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Superficial Safety Alignment HypothesisJianwei Li, Jung-Eun KimICLR 2026 · 11 citations
  2. Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean ReferenceJianwei Li, Jung-Eun KimICLR 2026 · 8 citations
  3. Safety Alignment of Large Language Models via Contrasting Safe and Harmful DistributionsXiaoyun Zhang, Zhengyue Zhao, Wenxuan Shi, Kaidi Xu et al.AAAI 2026 · 4 citations
  4. Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentMengxuan Hu, Vivek V. Datla, Anoop Kumar, Zihan Guan et al.ICLR 2026 · 3 citations
  5. New Wide-Net-Casting Jailbreak Attacks Risk Large ModelsQiuchi Xiang, Haoxuan Qu, Hossein Rahmani, Jun LiuICML 2026

Builds on26

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  5. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations

Related papers

  1. Resolving the Security-Auditability Dilemma with Auditable Latent Chain-of-Thought AlignmentGuan Wang, Biyu Zhou, Xuehai Tang, Jizhong Han et al.ACL 2026
  2. Safety Alignment Should be Made More Than Just a Few Tokens DeepXiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma et al.ICLR 2025
  3. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement LearningYi Zhang, An Zhang, XiuYu Zhang, Leheng Sheng et al.ICLR 2026 · 15 citations
  4. Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM SafetyYuyou Zhang, Miao Li, William Han, Yihang Yao et al.ACL Findings 2025 · 5 citations
  5. Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-DepthJiawei Zhang, Andrew Estornell, David D. Baek, Bo Li et al.ICLR 2026 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines