ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

The integration of large language models (LLMs) into a wide range of applications has highlighted the critical role of well-crafted system prompts, which require extensive testing and domain expertise.These prompts enhance task performance but may also encode sensitive information and filtering criteria, posing security risks if exposed.Recent research shows that system prompts are vulnerable to extraction attacks, while existing defenses are either easily bypassed or require constant updates to address new threats.In this work, we introduce ProxyPrompt, a novel defense mechanism that prevents prompt leakage by replacing the original prompt with a proxy.This proxy maintains the original task's utility while obfuscating the extracted prompt, ensuring attackers cannot reproduce the task or access sensitive information.Comprehensive evaluations on 264 LLM and system prompt pairs show that ProxyPrompt protects 94.70% of prompts from extraction attacks, outperforming the next-best defense, which only achieves 42.80%.* This work was conducted while Zhixiong and Irina were employed at the Bosch Center for Artificial Intelligence in Renningen, Germany.The code is available at https://github.com/boschresearch/proxyprompt.