Can Personal Health Information Be Secured in LLM? Privacy Attack and Defense in the Medical Domain

Recent advancements have shown that Large Language Models (LLMs) possess significant versatility, making them suitable for applications in many areas. Several studies have shown how general-purpose LLMs can be adapted to domain-specific tasks. However, these domain-adapted LLMs can be exposed to greater privacy risks, which are especially exacerbated in the medical field. In this paper, we present the study investigating the susceptibility of LLMs to leaking sensitive health information. We conduct prompt-based attacks on LLMs trained with medical datasets, showing that medical LLMs can inadvertently disclose confidential patient data. To contribute towards mitigating privacy risks in the medical domain, we implement red teaming defense strategies to make LLMs robust against malicious attacks. For this medical red teaming approach, we develop and publicly release MediRed, a dataset of 1,000 red team attacks. By leveraging this dataset to enhance our defense mechanisms, we achieve up to 56% improvement in privacy protection compared to base models. Our code and dataset are available at https://github.com/yujinKang32/Private_Med_LLM.git