EMNLP Findings2025
TS-SQL: Test-driven Self-refinement for Text-to-SQL
Wenbo Xu, Haifeng Zhu, Liang Yan, Chuanyi Liu, Peiyi Han, Shaoming Duan, Jeff Z. Pan
Abstract
Large Language Model (LLM)-based selfrefinement has advanced Text-to-SQL, but it struggles with SQL semantic errors, such as omitted conditions and misinterpreted requirements. This is because self-refinement depends on LLMs' semantic understanding of questions, a process prone to hallucination-induced biases, leading to uncorrectable errors. To solve this problem, we propose Test-driven Selfrefinement for Text-to-SQL (TS-SQL). It leverages a collaborative LLM agent framework to automatically synthesize high-quality test cases, including test data and test code. The test cases are further employed to provide execution feedback for LLM self-refinement towards SQL semantic errors. Rigorous evaluation shows the superiority of TS-SQL: for BIRD-dev, TS-SQL improves at least 6% over existing SQL self-refinement methods; for Spider-dev, TS-SQL identifies and corrects 131 gold SQL errors, exposing system flaws in benchmark rigor. For reproducibility, we release the modified Spider-dev benchmark to foster further research. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement cites Teaching Large Language Models to Self-Debug
- Teaching Large Language Models to Self-Debug cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Teaching Large Language Models to Self-Debug cites Self-Refine: Iterative Refinement with Self-Feedback
- Faith and Fate: Limits of Transformers on Compositionality cites Self-Refine: Iterative Refinement with Self-Feedback
Cited by top-tier papers1
Builds on21
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Teaching Large Language Models to Self-DebugXinyun Chen, Maxwell Lin, Nathanael Schärli, Denny ZhouICLR 2024 · 1,085 citations
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
- Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li et al.NeurIPS 2023 · 728 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
Related papers
- SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent CollaborationJipeng Cen, Jiaxin Liu, Zhixu Li, Jingjing WangAAAI 2025 · 18 citations
- GBV-SQL: Guided Generation and SQL2Text Back-Translation Validation for Multi-Agent Text2SQLDaojun Chen, Xi Wang, Shenyuan Ren, Qingzhi Ma et al.ACL 2026
- SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQLGeonho Lee, Min-Soo KimVLDB 2026
- ToolSQL: A Tool-Assisted Agent for SQL Verification and RefinementZhongyuan Wang, Richong Zhang, Zhijie Nie, Jaein KimKDD 2025 · 2 citations
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQLYue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya et al.NeurIPS 2025 · 21 citations
