AirGapAgent: Protecting Privacy-Conscious Conversational Agents
Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, Daniel Ramage
Abstract
The growing use of large language model (LLM)-based conversational agents to manage sensitive user data raises significant privacy concerns.While these agents excel at understanding and acting on context, this capability can be exploited by malicious actors.We introduce a novel threat model where adversarial third-party apps manipulate the context of interaction to trick LLM-based agents into revealing private information not relevant to the task at hand.Grounded in the framework of contextual integrity, we introduce AirGapAgent, a privacy-conscious agent designed to prevent unintended data leakage by restricting the agent's access to only the data necessary for a specific task.Extensive experiments using Gemini, GPT, and Mistral models as agents validate our approach's effectiveness in mitigating this form of context hijacking while maintaining core agent functionality.For example, we show that a single-query context hijacking attack on a Gemini Ultra agent reduces its ability to protect user data from 94% to 45%, while an AirGapAgent achieves 97% protection, rendering the same attack ineffective. CCS Concepts Security and privacy Information flow control.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
22 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Privacy Reasoning in Ambiguous Contexts cites Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
- Privacy Reasoning in Ambiguous Contexts cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- Contextual Integrity in LLMs via Reasoning and Reinforcement Learning cites Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy Management cites Toolformer: Language Models Can Teach Themselves to Use Tools
Cited by top-tier papers22
- Contextual Integrity in LLMs via Reasoning and Reinforcement LearningGuangchen Lan, Huseyin A. Inan, Sahar Abdelnabi, Janardhan Kulkarni et al.NeurIPS 2025 · 56 citations
- Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM AgentsZichuan Li, Jian Cui, Xiaojing Liao, Luyi XingNDSS 2026 · 24 citations
- Towards Automating Data Access Permissions in AI AgentsYuhao Wu, Ke Yang, Franziska Roesner, Tadayoshi Kohno et al.S&P 2026 · 22 citations
- Privacy Reasoning in Ambiguous ContextsRen Yi, Octavian Suciu, Adrià Gascón, Sarah Meiklejohn et al.NeurIPS 2025 · 15 citations
- From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy ManagementEryue Xu, Tianshi LiCHI 2026 · 3 citations
Builds on10
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Red Teaming Language Models with Language ModelsEthan Perez, Saffron Huang, H. Francis Song, Trevor Cai et al.EMNLP 2022 · 239 citations
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity TheoryNiloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov et al.ICLR 2024 · 198 citations
- Fundamental Limitations of Alignment in Large Language ModelsYotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine et al.ICML 2024 · 186 citations
- TAG: Gradient Attack on Transformer-based Language ModelsJieren Deng, Yijue Wang, Ji Li, Chenghong Wang et al.EMNLP Findings 2021 · 59 citations
Related papers
- Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational AgentsIvoline C. Ngong, Swanand Ravindra Kadhe, Hao Wang, Keerthiram Murugesan et al.ACL Findings 2025 · 7 citations
- PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference TrainingYuhan Cheng, Hancheng Ye, Hai Li, Jingwei Sun et al.ICML 2026 · 3 citations
- Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered AgentsShouju Wang, Fenglin Yu, Xirui Liu, Xiaoting Qin et al.EMNLP Findings 2025 · 7 citations
- Unveiling Privacy Risks in LLM Agent MemoryBo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang et al.ACL 2025
- PrivGate: Steering Contextual Integrity in LLMs via Latent Space GeometryRunshan Hu, Yukun Dong, Yingying Huangfu, Ruohan Zhao et al.ICML 2026
