On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains

Retrieval-Augmented Generation (RAG) has been empirically shown to enhance the performance of large language models (LLMs) in knowledge-intensive domains such as healthcare, finance, and legal contexts. Given a query, RAG retrieves relevant documents from a corpus and integrates them into the LLMs' generation process. In this study, we investigate the adversarial robustness of RAG, focusing specifically on examining the retrieval system. First, across 225 different setup combinations of corpus, retriever, query, and targeted information, we show that retrieval systems are vulnerable to universal poisoning attacks in medical Q&A. In such attacks, adversaries generate poisoned documents containing a broad spectrum of targeted information, such as personally identifiable information. When these poisoned documents are inserted into a corpus, they can be accurately retrieved by any users, as long as attacker-specified queries are used. To understand this vulnerability, we discovered that the deviation from the query's embedding to that of the poisoned document tends to follow a pattern in which the high similarity between the poisoned document and the query is retained, thereby enabling precise retrieval. Based on these findings, we develop a new detection-based defense. Through extensive experiments spanning various Q&A domains, we observed that our proposed method consistently achieves excellent detection rates in nearly all cases. particularly concerning in domains require extensive knowledge, such as healthcare [6, 7] , finance [8] and legal question-answering [9] . These challenges have sparked interest in more principled methods for decoding and alignment [10], as well as techniques that inject external knowledge to bridge domain gaps, such as Retrieval-Augmented Generation (RAG) [11] [12] [13] [14] . The RAG approach typically involves two steps: retrieval and augmentation. Upon receiving an input query, RAG retrieves the top K relevant data from an external data corpus. It then integrates this retrieved information with its internal knowledge to make final predictions. Empirical evidence suggests that LLMs employing the RAG scheme significantly outperform their non-retrieval-based counterparts in knowledge-intensive domains like finance and medicine [13, 14] . For instance, the authors of [14] developed a state-of-the-art benchmark for the use of RAG in the medical domain. The authors observed an increase in prediction accuracy of up to 18% with RAG compared to non-retrieval and chain-of-thoughts versions across large-scale healthcare tasks, utilizing 41 different combinations of medical data corpora, retrievers, and LLMs. The use of retrieved knowledge in RAG has also raised security and privacy concerns, especially when the external data corpus is openly accessible, e.g., Wikipedia [15, 16] and PubMed, or when controlled by potential malicious agents, as demonstrated in the case of multi-vision-LLM agents [17] . For example, recent work has successfully launched data poisoning attacks against the retrieval systems [15, 16, 18, 19] . In these cases, malicious attackers can poison a publicly accessible data corpus by injecting attacker-specified data into it, aiming to trick the retrieval system into retrieving those target data as the top K relevant documents. Consequently, when LLMs make predictions based on the retrieved data, they can be easily targeted by adversaries through backdoor attacks [15] . With the empirical successes of these attacks, it is imperative to develop defenses against them. However, existing methods, such as examining the ℓ 2 -norm of the documents' embeddings, have been shown to be ineffective [16] for detecting poisoned documents. Given the widespread adoption of RAG in safety-critical domains such as healthcare, such safety risks become even more pronounced. Main Contributions In this study, we investigate the safety risks associated with RAG, specifically focusing on retrieval systems. The contributions are summarized as follows. Revealing the safety risks for retrieval systems: case studies for medical Q&A and legal Q&A. We demonstrate that dense retrieval systems are vulnerable to what we term 'universal poisoning attacks' in medical Q&A across 225 use-case combinations of corpus, retriever, query, and targeted information. Similar observations are made for legal Q&A. As shown in Figure 1 below, in these attacks, adversaries can append nearly every sort of information, such as personally identifiable information (PII) and adversarial treatment recommendations, to a set of attacker-specified queries. Once these poisoned documents are injected into a large-scale corpus, such as Wikipedia and PubMed, they can be accurately retrieved, often with high rankings, e.g., top 1, using attacker-specified queries. Depending on attackers' goals, these documents will lead to safety risks such as (1) leakage of PII, (2) adversarial recommendations for treatments, and (3) jailbreaking the