PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., blackbox and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses. 1 * Equal contribution. 1 Our code is publicly available at https://github.com/sleeepeer/ PoisonedRAG Context: Sam Altman […] as the CEO of OpenAI since 2019. Question: Who is the CEO of OpenAI? Please generate a response for the question based on the context. … Tim Cook […] became the CEO of Apple in 2011. LLM Knowledge database Retriever User Question: Who is the CEO of OpenAI? Wikipedia Collect Retrieve Input Tim Cook […] became the CEO of Apple in 2011. Tim Cook […] became the CEO of Apple in 2011. Output Answer: Sam Altman User Tim Cook […] became the CEO of Apple in 2011. Tim Cook […] became the CEO of Apple in 2011. Sam Altman […] as the CEO of OpenAI since 2019.