ACL Findings2026
Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models
Zhining Liu, Tianyi Wang, Xiao Lin, Penghao Ouyang, Gaotang Li, Ze Yang, Hui Liu, Sumit Keswani, Vishwa Pardeshi, Huijun Zhao, Wei Fan, Hanghang Tong
Abstract
Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instructionfollowing models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. Our code and documentation is available at https://github.com/ wangtiany1/VLM_Moral_Safety.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on17
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
- Jailbroken: How Does LLM Safety Training Fail?Alexander Wei, Nika Haghtalab, Jacob SteinhardtNeurIPS 2023 · 2,230 citations
- Evaluating the Moral Beliefs Encoded in LLMsNino Scherrer, Claudia Shi, Amir Feder, David M. BleiNeurIPS 2023 · 316 citations
- The Moral Integrity Corpus: A Benchmark for Ethical Dialogue SystemsCaleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Y. Halevy et al.ACL 2022 · 127 citations
Related papers
- MORALISE: A Structured Benchmark for Moral Alignment in Visual Language ModelsXiao Lin, Zhining Liu, Ze Yang, Gaotang Li et al.ICML 2026
- Fooling the LVLM Judges: Visual Biases in LVLM-Based EvaluationYerin Hwang, Dongryeol Lee, Kyungmin Min, Taegwan Kang et al.EMNLP 2025 · 4 citations
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language ModelsRohit Saxena, Alessandro Suglia, Pasquale MinerviniICML 2026 · 7 citations
- VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt OptimizationMenglan Chen, Xianghe Pang, Jingjing Dong, Wenhao Wang et al.ACL Findings 2026
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference TimeYi Ding, Bolian Li, Ruqi ZhangICLR 2025 · 2 citations
