ACL Findings2024
Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL translation
Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto
Abstract
Understanding textual description to generate code seems to be an achieved capability of instruction-following Large Language Models (LLMs) in zero-shot scenario. However, there is a severe possibility that this translation ability may be influenced by having seen target textual descriptions and the related code. This effect is known as Data Contamination.
In this study, we investigate the impact of Data Contamination on the performance of GPT-3.5 in the Text-to-SQL code-generating tasks. Hence, we introduce a novel method to detect Data Contamination in GPTs and examine GPT-3.5's Text-to-SQL performances using the known Spider Dataset and our new unfamiliar dataset Termite. Furthermore, we analyze GPT-3.5's efficacy on databases with modified information via an adversarial table disconnection (ATD) approach, complicating Textto-SQL tasks by removing structural pieces of information from the database. Our results indicate a significant performance drop in GPT-3.5 on the unfamiliar Termite dataset, even with ATD modifications, highlighting the effect of Data Contamination on LLMs in Text-to-SQL translation tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models cites Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models cites Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites Semantic Evaluation for Text-to-SQL with Distilled Test Suites
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation cites Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation cites Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation cites MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation cites Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Cited by top-tier papers8
- MMLU-CF: A Contamination-free Multi-task Language Understanding BenchmarkQihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui et al.ACL 2025 · 33 citations
- Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language ModelsElena Sofia Ruzzetti, Giancarlo A. Xompero, Davide Venditti, Fabio Massimo ZanzottoACL 2025 · 9 citations
- On The Fragility of Benchmark Contamination Detection in Reasoning ModelsHan Wang, Haoyu Li, Brian Ko, Huan ZhangICLR 2026 · 8 citations
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic EvaluationSimin Chen, Yiming Chen, Zexin Li, Yifan Jiang et al.EMNLP 2025 · 2 citations
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language ModelsMartin Riddell, Ansong Ni, Arman CohanACL 2024
Builds on6
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
- Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationDawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun et al.VLDB 2024 · 609 citations
- Time Travel in LLMs: Tracing Data Contamination in Large Language ModelsShahriar Golchin, Mihai SurdeanuICLR 2024 · 165 citations
- Semantic Evaluation for Text-to-SQL with Distilled Test SuitesRuiqi Zhong, Tao Yu, Dan KleinEMNLP 2020 · 88 citations
- Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4Kent K. Chang, Mackenzie Cramer, Sandeep Soni, David BammanEMNLP 2023 · 70 citations
Related papers
- Contamination Means Overestimation? A Fine-Grained Empirical Study in Code IntelligenceZhen Yang, Hongyi Lin, Yifan He, Junqi Wang et al.ISSTA 2026
- SQL-Checker: Error Detection and Labeling for Text-to-SQL with Interpretability AnalysisXingyu Ma, Xin Tian, Lingxiang Wu, Xuepeng Wang et al.WWW 2026
- Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing ComparisonShuo Sun, Yuchen Zhang, Jiahuan Yan, Yuze Gao et al.EMNLP Findings 2023 · 9 citations
- CodeS: Towards Building Open-source Language Models for Text-to-SQLHaoyang Li, Jing Zhang, Hanbing Liu, Ju Fan et al.SIGMOD 2024 · 124 citations
- Cracking Query Bottlenecks: Towards Efficiency-Oriented Text-to-SQL GenerationLi Lin, Yunfeng Shen, Lingfeng Bao, Rongxin Wu et al.ISSTA 2026
