Lune

EMNLP Findings2024

Can Textual Unlearning Solve Cross-Modality Safety Alignment?

Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael B. Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit K. Roy-Chowdhury, Chengyu Song

Abstract

Content warning: This paper contains unsafe model-generated content.

Recent studies reveal that integrating new modalities into large language models (LLMs), such as vision-language models (VLMs), creates a new attack surface that bypasses existing safety training techniques like supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). While further SFT and RLHF-based safety training can be conducted in multi-modal settings, collecting multi-modal training datasets poses a significant challenge. Inspired by the structural design of recent multi-modal models, where all input modalities are ultimately fused into the language space, we explore whether unlearning solely in the textual domain can be effective for cross-modality safety alignment. Our empirical evaluation across seven datasets demonstrates promising transferability -textual unlearning in VLMs significantly reduces the Attack Success Rate (ASR) to less than 8% and in some cases, even as low as nearly 2% for both textbased and vision-text-based attacks, alongside preserving the utility. Moreover, our experiments show that unlearning with a multi-modal dataset offers no potential benefits but incurs significantly increased computational demands.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • VL-Eraser: Vacuum Distillation for Machine Unlearning in Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • VL-Eraser: Vacuum Distillation for Machine Unlearning in Vision-Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Finetuned Language Models are Zero-Shot Learners
  • Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots cites LoRA: Low-Rank Adaptation of Large Language Models
  • Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots cites QLoRA: Efficient Finetuning of Quantized LLMs
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind SpotsErfan Shayegani, G. M. Shahariar, Sara Abdali, Lei Yu et al.ICLR 2026 · 3 citations
  2. VL-Eraser: Vacuum Distillation for Machine Unlearning in Vision-Language ModelsYili Wang, Lu Dai, Tairan Huang, Yijie Xu et al.CVPR 2026
  3. Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language ModelsSaketh Bachu, Erfan Shayegani, Rohit Lal, Trishna Chakraborty et al.ICML 2025

Builds on33

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  5. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations

Related papers

  1. Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine UnlearningYiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen et al.ICLR 2026
  2. VLSBench: Unveiling Visual Leakage in Multimodal SafetyXuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang et al.ACL 2025
  3. Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?Yanbo Wang, Jiyang Guan, Jian Liang, Ran HeCVPR 2025
  4. Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related ImagesQishun Yang, Shu Yang, Lijie Hu, Di WangACL 2026 · 1 citation
  5. Unraveling and Mitigating Safety Alignment Degradation of Vision-Language ModelsQin Liu, Chao Shang, Ling Liu, Nikolaos Pappas et al.ACL Findings 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines