DocMIA: Document-Level Membership Inference Attacks against DocVQA Models
Khanh Nguyen, Raouf Kerkouche, Mario Fritz, Dimosthenis Karatzas
Abstract
Document Visual Question Answering (DocVQA) has introduced a new paradigm for end-to-end document understanding, and quickly became one of the standard benchmarks for multimodal LLMs. Automating document processing workflows, driven by DocVQA models, presents significant potential for many business sectors. However, documents tend to contain highly sensitive information, raising concerns about privacy risks associated with training such DocVQA models. One significant privacy vulnerability, exploited by the membership inference attack, is the possibility for an adversary to determine if a particular record was part of the model's training data. In this paper, we introduce two novel membership inference attacks tailored specifically to DocVQA models. These attacks are designed for two different adversarial scenarios: a white-box setting, where the attacker has full access to the model architecture and parameters, and a black-box setting, where only the model's outputs are available. Notably, our attacks assume the adversary lacks access to auxiliary datasets, which is more realistic in practice but also more challenging. Our unsupervised methods outperform existing state-of-the-art membership inference attacks across a variety of DocVQA models and datasets, demonstrating their effectiveness and highlighting the privacy risks in this domain.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Membership Inference Attacks Against Machine Learning Models cites Deep Learning with Differential Privacy
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning cites Membership Inference Attacks Against Machine Learning Models
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning cites ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning cites Deep Learning with Differential Privacy
- ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models cites Membership Inference Attacks Against Machine Learning Models
Builds on24
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan et al.CCS 2016 · 7,620 citations
- Membership Inference Attacks Against Machine Learning ModelsReza Shokri, Marco Stronati, Congzheng Song, Vitaly ShmatikovS&P 2017 · 5,137 citations
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated LearningMilad Nasr, Reza Shokri, Amir HoumansadrS&P 2019 · 1,778 citations
- ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning ModelsAhmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang et al.NDSS 2019 · 1,141 citations
Related papers
- LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language ModelsYihao Liu, Xinqi Lyu, Dong Wang, Yanjie Li et al.NeurIPS 2025 · 3 citations
- Order of Magnitude Speedups for LLM Membership InferenceRongting Zhang, Martin Bertran Lopez, Aaron RothEMNLP 2024 · 1 citation
- Quantifying Privacy Risks of Masked Language Models Using Membership Inference AttacksFatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick et al.EMNLP 2022 · 72 citations
- Membership Inference Attacks Against Vision-Language ModelsYuke Hu, Zheng Li, Zhihao Liu, Yang Zhang et al.USENIX Security 2025
- Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory ProbingJinhua Yin, Peiru Yang, Chen Yang, Huili Wang et al.NeurIPS 2025 · 4 citations
