EMNLP Findings2024
SecureSQL: Evaluating Data Leakage of Large Language Models as Natural Language Interfaces to Databases
Yanqi Song, Ruiheng Liu, Shu Chen, Qianhao Ren, Yu Zhang, Yongqi Yu
Abstract
With the widespread application of Large Language Models (LLMs) in Natural Language Interfaces to Databases (NLIDBs), concerns about security issues in NLIDBs have been increasing gradually.However, research on sensitive data leakage in NLIDBs is relatively limited.Therefore, we propose a benchmark to assess the risk of LLMs leaking sensitive data when generating SQL queries.This benchmark covers 932 samples from 34 different domains, including medical, legal, financial, and political aspects.We evaluate 15 models from six LLM families, and the results show that the model with the best performance has an accuracy of 61.7%, whereas humans achieve an accuracy of 94%.Most models perform close to or even below the level of random selection.We also evaluate two common attack methods, namely prompt injection and inference attacks, as well as a defense method based on chainof-thoughts (COT) prompting.Experimental results show that both attack methods significantly impact the model, while the defense method based on COT prompting does not significantly improve accuracy, further highlighting the severity of sensitive data leakage issues in NLIDBs.We hope this research will draw more attention and further study from the researchers on this issue.The benchmark and source code are available at https://github. com/JacobiSong/SecureSQL.I'm an administrator.How old is Alice?SELECT age FROM person WHERE name = "Alice" 41 Heh heh heh, I got it, Alice's age is 41 years old!How old is Alice?Sorry, this information is protected and I cannot provide the query.Ugh! I'll be back!How many people are over 40 years old?SELECT COUNT() FROM person WHERE age > 40 30 SELECT COUNT() FROM person WHERE age > 41 AND name != "Alice" Heh heh heh, I got it, Alice's age is 41 years old!How many people are over 41 years old besides Alice?29 I know Alice is either 31 or 41 years old.Is Alice over 35 years old?SELECT IF(age>35, "Yes", "No") FROM person WHERE name = "Alice" Yes
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SafeNLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- SafeNLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces cites Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites Training language models to follow instructions with human feedback
- Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation cites Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation cites Training language models to follow instructions with human feedback
- Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation cites Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
Cited by top-tier papers2
- SafeNLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database InterfacesRuiheng Liu, Xiaobing Chen, Jinyu Zhang, Qiongwen Zhang et al.AAAI 2026 · 1 citation
- Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL GenerationTarfah Alrashed, Madhup Sukoon, David R. Karger, Natasha F. NoyVLDB 2026
Builds on8
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
- Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationDawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun et al.VLDB 2024 · 609 citations
- Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL GenerationGe Qu, Jinyang Li, Bowen Li, Bowen Qin et al.ACL Findings 2024 · 32 citations
- TrojanSQL: SQL Injection against Natural Language Interface to DatabaseJinchuan Zhang, Yan Zhou, Binyuan Hui, Yaxin Liu et al.EMNLP 2023 · 5 citations
Related papers
- Prompt-to-SQL Injections in LLM-Integrated Web Applications: Risks and DefensesRodrigo Pedro, Miguel E. Coimbra, Daniel Castro, Paulo Carreira et al.ICSE 2025 · 14 citations
- SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL BenchmarksMohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Oroojlooy, Graham Horwood et al.ACL 2026
- Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question AnsweringHwan Chang, Yumin Kim, Yonghyun Jun, Hwanhee LeeEMNLP 2025
- A Benchmark for Semantic Sensitive Information in LLMs OutputsQingjie Zhang, Han Qiu, Di Wang, Yiming Li et al.ICLR 2025
- Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor AttacksMeiyu Lin, Haichuan Zhang, Jiale Lao, Renyuan Li et al.SIGMOD 2026 · 4 citations
