ACL Findings2026
Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models
Haeun Jang, Hwan Chang, Hwanhee Lee
Abstract
The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering is often constrained by dynamic, user-defined policies that dictate information disclosure based on context. While ensuring adherence to these explicit constraints is critical, existing safety research primarily focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents. In this paper, we introduce Doc-PP (Document Policy Preservation Benchmark), a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies. Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities, effectively circumventing existing safety constraints. Furthermore, we identify that providing extracted text improves perception but inadvertently facilitates leakage. To address these vulnerabilities, we propose DVA (Decompose-Verify-Aggregation), a structural inference framework that decouples reasoning from policy verification. Experimental results demonstrate that DVA significantly outperforms standard prompting defenses, offering a robust baseline for policy-compliant document understanding.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- Do Vision-Language Models Respect Contextual Integrity in Location Disclosure? cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
Builds on5
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity TheoryNiloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov et al.ICLR 2024 · 198 citations
- M-LongDoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning FrameworkYew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song et al.EMNLP 2025 · 3 citations
- CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistsYukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho et al.EMNLP 2025 · 2 citations
- Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?Ruixin Yang, Ethan Mendes, Arthur Wang, James Hays et al.ICLR 2026 · 1 citation
- Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question AnsweringHwan Chang, Yumin Kim, Yonghyun Jun, Hwanhee LeeEMNLP 2025
Related papers
- VLSBench: Unveiling Visual Leakage in Multimodal SafetyXuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang et al.ACL 2025
- Mitigating Safety Context Amnesia in Multimodal Reasoning Models via Intent-Guided Safety ReasoningXiyao Dong, Guangsheng Cheng, YiLong Chen, Xiaojin Zhang et al.ACL 2026
- Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAGuanhua Chen, Yutong Yao, Shenghe Sun, Ci-Jun Gao et al.ACL Findings 2026
- DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense DocumentsZhuoran Yu, Le T Nguyen, Jaden Park, Xinyi Gu et al.ICML 2026
- DR-VQA: Decompose-then-Reconstruct for Visual Question Answering in BLV AssistanceBocheng Pan, Hailong Shi, Xingyu GaoACM MM 2025
