ACL Findings2025
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluís Màrquez, Miguel Ballesteros, Yassine Benajiba
Abstract
The safety alignment ability of Vision-Language Models (VLMs) is prone to be degraded by the integration of the vision module compared to its LLM backbone. We investigate this phenomenon, dubbed as "safety alignment degradation" in this paper, and show that the challenge arises from the representation gap that emerges when introducing vision modality to VLMs. In particular, we show that the representations of multi-modal inputs shift away from those of text-only inputs that represent the distribution that the LLM backbone is optimized for. At the same time, the safety alignment capabilities, initially developed within the textual embedding space, do not successfully transfer to this new multi-modal representation space. To reduce safety alignment degradation, we introduce Cross-Modality Representation Manipulation (CMRM), an inference-time representation intervention method for recovering the safety alignment ability that is inherent in the LLM backbone of VLMs, while simultaneously preserving the functional capabilities of VLMs. The empirical results show that our framework significantly recovers the alignment ability that is inherited from the LLM backbone with minimal impact on the fluency and linguistic capabilities of pre-trained VLMs even without additional training. Specifically, the unsafe rate of LLaVA-7B on multi-modal input can be reduced from 61.53% to as low as 3.15% with only inference-time intervention.
WARNING: This paper contains examples of toxic or harmful language.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
15 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Unveiling the Basin-Like Loss Landscape in Large Language Models cites Training language models to follow instructions with human feedback
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Unveiling the Basin-Like Loss Landscape in Large Language Models
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation cites Training language models to follow instructions with human feedback
- Understanding and Rectifying Safety Perception Distortion in VLMs cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers15
- BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled OptimizationXueyang Zhou, Guiyao Tie, Guowen Zhang, Hechang Wang et al.NeurIPS 2025 · 50 citations
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning PerturbationYibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao et al.NeurIPS 2025 · 22 citations
- Understanding and Rectifying Safety Perception Distortion in VLMsXiaohan Zou, Jian Kang, George Kesidis, Lu LinNeurIPS 2025 · 20 citations
- Unveiling the Basin-Like Loss Landscape in Large Language ModelsHuanran Chen, Zeming Wei, Yao Huang, Yichi Zhang et al.ICLR 2026 · 14 citations
- Attention! Your Vision Language Model Could Be Maliciously ManipulatedXiaosen Wang, Shaokang Wang, Zhijin Ge, Yuyang Luo et al.NeurIPS 2025 · 13 citations
Builds on14
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
Related papers
- How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee et al.ICLR 2025
- Can Textual Unlearning Solve Cross-Modality Safety Alignment?Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael B. Abu-Ghazaleh et al.EMNLP Findings 2024 · 3 citations
- Cross-Modal Safety Mechanism Transfer in Large Vision-Language ModelsShicheng Xu, Liang Pang, Yunchang Zhu, Huawei Shen et al.ICLR 2025
- MMAligner: Safeguarding Multimodal Large Language Models through Representation CalibrationShenyi Zhang, Keyan Guo, Zihao Wang, Xuebin Li et al.CCS 2026
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?Yanbo Wang, Jiyang Guan, Jian Liang, Ran HeCVPR 2025
