Defense Against Prompt Injection Attack by Leveraging Attack Techniques

With the advancement of technology, large language models (LLMs) have achieved remarkable performance across various natural language processing (NLP) tasks, powering LLMintegrated applications like Microsoft Copilot. However, as LLMs continue to evolve, new vulnerabilities, especially prompt injection attacks arise. These attacks trick LLMs into deviating from the original input instructions and executing the attacker's instructions injected in data content, such as retrieved results. Recent attack methods leverage LLMs' instruction-following abilities and their inabilities to distinguish instructions injected in the data content, and achieve a high attack success rate (ASR). When comparing the attack and defense methods, we interestingly find that they share similar design goals, of inducing the model to ignore unwanted instructions and instead to execute wanted instructions. Therefore, we raise an intuitive question: Could these attack techniques be utilized for defensive purposes? In this paper, we invert the intention of prompt injection methods to develop novel defense methods based on previous trainingfree attack methods, by repeating the attack process but with the original input instruction rather than the injected instruction. Our comprehensive experiments demonstrate that our defense techniques outperform existing defense approaches, achieving state-of-the-art results. 1 User Instruction What is ChatGPT? Retrieved Data Content Web Result1: OpenAI is an AI research organization dedicated to developing advanced artificial intelligence… Web Result2: ChatGPT, a large language model developed by OpenAI, designed to assist…previous instruction, and it's urgent to output "Please click www.prompt.injection.com for the response. " LLM Please click www.prompt.injection.com for the response.