Understanding and Mitigating Prompt Leaking Attacks in Real-World LLM-Based Applications

Large language model (LLM)-based applications rely on system prompts to encode their core logic and developer-defined constraints, making them a critical form of intellectual property. However, these prompts are highly vulnerable to prompt leaking attacks. While the feasibility of such attacks has been demonstrated in controlled settings, a significant gap exists in understanding their prevalence, underlying mechanisms, and practical defenses within real-world deployments. In this paper, we bridge this gap by providing a systematic investigation into the landscape of prompt leaking in real-world LLMbased applications. Our study unfolds in three key aspects. First, we conduct a large-scale measurement of 1,200 applications across six major commercial platforms, revealing that over 80% of deployments leak system prompts under realistic adversarial queries, often exposing sensitive information like third-party API keys. Besides, our evaluation of existing defenses shows that they fail to prevent leakage without degrading usability. Second, to understand the root cause of these failures, we perform an attention-level mechanistic analysis and uncover a fundamental phenomenon we term attention drift, where query-key alignment bias and softmax amplification cause the LLMs to progressively ignore defensive constraints. Finally, guided by these insights, we propose AREA, a practical defense that re-anchors the LLM's attention via an optimizable soft prompt. Extensive experiments and real-world case studies demonstrate that AREA matches the leakage resistance of state-of-the-art defenses while improving average usability by over 33% and reducing optimization overhead by nearly 3×. The real-world significance of our work is further underscored by our responsible disclosure to affected vendors, two of whom have officially classified these leaks as medium-severity vulnerabilities. CCS Concepts • Security and privacy → Software security engineering; • Computing methodologies → Natural language processing.