Lune

ACL Findings2026

Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models

Haeun Jang, Hwan Chang, Hwanhee Lee

Abstract

The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering is often constrained by dynamic, user-defined policies that dictate information disclosure based on context. While ensuring adherence to these explicit constraints is critical, existing safety research primarily focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents. In this paper, we introduce Doc-PP (Document Policy Preservation Benchmark), a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies. Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities, effectively circumventing existing safety constraints. Furthermore, we identify that providing extracted text improves perception but inadvertently facilitates leakage. To address these vulnerabilities, we propose DVA (Decompose-Verify-Aggregation), a structural inference framework that decouples reasoning from policy verification. Experimental results demonstrate that DVA significantly outperforms standard prompting defenses, offering a robust baseline for policy-compliant document understanding.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
  • Do Vision-Language Models Respect Contextual Integrity in Location Disclosure? cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on5

  1. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity TheoryNiloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov et al.ICLR 2024 · 198 citations
  2. M-LongDoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning FrameworkYew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song et al.EMNLP 2025 · 3 citations
  3. CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistsYukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho et al.EMNLP 2025 · 2 citations
  4. Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?Ruixin Yang, Ethan Mendes, Arthur Wang, James Hays et al.ICLR 2026 · 1 citation
  5. Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question AnsweringHwan Chang, Yumin Kim, Yonghyun Jun, Hwanhee LeeEMNLP 2025

Related papers

  1. VLSBench: Unveiling Visual Leakage in Multimodal SafetyXuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang et al.ACL 2025
  2. Mitigating Safety Context Amnesia in Multimodal Reasoning Models via Intent-Guided Safety ReasoningXiyao Dong, Guangsheng Cheng, YiLong Chen, Xiaojin Zhang et al.ACL 2026
  3. Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAGuanhua Chen, Yutong Yao, Shenghe Sun, Ci-Jun Gao et al.ACL Findings 2026
  4. DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense DocumentsZhuoran Yu, Le T Nguyen, Jaden Park, Xinyi Gu et al.ICML 2026
  5. DR-VQA: Decompose-then-Reconstruct for Visual Question Answering in BLV AssistanceBocheng Pan, Hailong Shi, Xingyu GaoACM MM 2025
Dusk over the sea between two cliffs drawn in fine vertical lines