Lune

EMNLP Findings2025

Tracing Training Footprints: A Calibration Approach for Membership Inference Attacks Against Multimodal Large Language Models

Xiaofan Zheng, Huixuan Zhang, Xiaojun Wan

Abstract

With the increasing scale of training data for Multimodal Large Language Models (MLLMs) and the lack of data details, there is growing concern about privacy breaches and data security issues. Under black-box access, exploring effective Membership Inference Attacks (MIA) has garnered increasing attention. In real-world applications, where most samples are non-members, the issue of non-members being over-represented in the data manifold, leading to misclassification as member samples, becomes more prominent. This has motivated recent work to focus on developing effective difficulty calibration strategies, producing promising results. However, these methods only consider text-only input during calibration, and their effectiveness is diminished when migrated to MLLMs due to the presence of visual embeddings. To address the above problem, we propose PC-MMIA, focusing on visual instruction fine-tuning data. PC-MMIA is based on the idea that tokens located in poorly generalized local manifolds can better reflect traces of member samples that have been trained. By employing bidirectional perturbation of image embeddings to capture tokens critical to MIA and assigning them different weights, we achieve difficulty calibration. Experimental results demonstrate that our proposed method surpasses existing methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Proving Test Set Contamination in Black-Box Language Models cites Membership Inference Attacks Against Machine Learning Models
  • MM-LLMs: Recent Advances in MultiModal Large Language Models cites Visual Instruction Tuning
  • UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint cites Visual Instruction Tuning
  • Detecting Pretraining Data from Large Language Models cites Membership Inference Attacks Against Machine Learning Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization FingerprintXiaofan Zheng, Xinghao Wang, Xiaojun WanACL 2026

Builds on22

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. Membership Inference Attacks Against Machine Learning ModelsReza Shokri, Marco Stronati, Congzheng Song, Vitaly ShmatikovS&P 2017 · 5,137 citations
  3. Detecting Pretraining Data from Large Language ModelsWeijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang et al.ICLR 2024 · 365 citations
  4. MM-LLMs: Recent Advances in MultiModal Large Language ModelsDuzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li et al.ACL Findings 2024 · 260 citations
  5. Proving Test Set Contamination in Black-Box Language ModelsYonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak et al.ICLR 2024 · 220 citations

Related papers

  1. Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory ProbingJinhua Yin, Peiru Yang, Chen Yang, Huili Wang et al.NeurIPS 2025 · 4 citations
  2. Membership Inference Attacks against Large Vision-Language ModelsZhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin et al.NeurIPS 2024 · 43 citations
  3. On the Importance of Difficulty Calibration in Membership Inference AttacksLauren Watson, Chuan Guo, Graham Cormode, Alexandre SablayrollesICLR 2022 · 189 citations
  4. LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language ModelsYihao Liu, Xinqi Lyu, Dong Wang, Yanjie Li et al.NeurIPS 2025 · 3 citations
  5. DF-MIA: A Distribution-Free Membership Inference Attack on Fine-Tuned Large Language ModelsZhiheng Huang, Yannan Liu, Daojing He, Yu LiAAAI 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines