Lune

EMNLP Findings2024

Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models

Xu Han, Linghao Jin, Xuezhe Ma, Xiaofeng Liu

Abstract

Fine-tuning pre-trained Vision-Language Models (VLMs) has shown remarkable capabilities in medical image and textual depiction synergy. Nevertheless, many pre-training datasets are restricted by patient privacy concerns, potentially containing noise that can adversely affect downstream performance. Moreover, the growing reliance on multi-modal generation exacerbates this issue because of its susceptibility to adversarial attacks. To investigate how VLMs trained on adversarial noisy data perform on downstream medical tasks, we first craft noisy upstream datasets using multi-modal adversarial attacks. Through our comprehensive analysis, we unveil that moderate noise enhances model robustness and transferability, but increasing noise levels negatively impact downstream task performance. To mitigate this issue, we propose rectify adversarial noise (RAN) framework, a recipe designed to effectively defend adversarial attacks and rectify the influence of upstream noise during fine-tuning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Progressive Compositionality in Text-to-Image Generative Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Progressive Compositionality in Text-to-Image Generative ModelsXu Han, Linghao Jin, Xiaofeng Liu, Paul Pu LiangICLR 2025

Builds on18

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  3. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  4. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised LearningAdrien Bardes, Jean Ponce, Yann LeCunICLR 2022 · 1,226 citations
  5. Diffusion Models for Adversarial PurificationWeili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao et al.ICML 2022 · 663 citations

Related papers

  1. Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-Based AttacksJiawei Wang, Yushen Zuo, Yuanjun Chai, Zhendong Liu et al.ICCV 2025
  2. When Background Matters: Breaking Medical Vision Language Models by Transferable AttackAkash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying ChenACL 2026
  3. VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained ModelsZiyi Yin, Muchao Ye, Tianrong Zhang, Jiaqi Wang et al.AAAI 2024 · 20 citations
  4. Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language ModelsFuta Waseda, Saku Sugawara, Isao EchizenACM MM 2025 · 2 citations
  5. VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained ModelsZiyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du et al.NeurIPS 2023 · 109 citations
Dusk over the sea between two cliffs drawn in fine vertical lines