Lune

EMNLP Findings2025

TS-SQL: Test-driven Self-refinement for Text-to-SQL

Wenbo Xu, Haifeng Zhu, Liang Yan, Chuanyi Liu, Peiyi Han, Shaoming Duan, Jeff Z. Pan

Abstract

Large Language Model (LLM)-based selfrefinement has advanced Text-to-SQL, but it struggles with SQL semantic errors, such as omitted conditions and misinterpreted requirements. This is because self-refinement depends on LLMs' semantic understanding of questions, a process prone to hallucination-induced biases, leading to uncorrectable errors. To solve this problem, we propose Test-driven Selfrefinement for Text-to-SQL (TS-SQL). It leverages a collaborative LLM agent framework to automatically synthesize high-quality test cases, including test data and test code. The test cases are further employed to provide execution feedback for LLM self-refinement towards SQL semantic errors. Rigorous evaluation shows the superiority of TS-SQL: for BIRD-dev, TS-SQL improves at least 6% over existing SQL self-refinement methods; for Spider-dev, TS-SQL identifies and corrects 131 gold SQL errors, exposing system flaws in benchmark rigor. For reproducibility, we release the modified Spider-dev benchmark to foster further research. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
  • ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement cites Teaching Large Language Models to Self-Debug
  • Teaching Large Language Models to Self-Debug cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Teaching Large Language Models to Self-Debug cites Self-Refine: Iterative Refinement with Self-Feedback
  • Faith and Fate: Limits of Transformers on Compositionality cites Self-Refine: Iterative Refinement with Self-Feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ErrorLLM: Modeling SQL Errors for Text-to-SQL RefinementZijin Hong, Hao Chen, Zheng Yuan, Qinggang Zhang et al.KDD 2026 · 3 citations

Builds on21

  1. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  2. Teaching Large Language Models to Self-DebugXinyun Chen, Maxwell Lin, Nathanael Schärli, Denny ZhouICLR 2024 · 1,085 citations
  3. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
  4. Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li et al.NeurIPS 2023 · 728 citations
  5. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations

Related papers

  1. SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent CollaborationJipeng Cen, Jiaxin Liu, Zhixu Li, Jingjing WangAAAI 2025 · 18 citations
  2. GBV-SQL: Guided Generation and SQL2Text Back-Translation Validation for Multi-Agent Text2SQLDaojun Chen, Xi Wang, Shenyuan Ren, Qingzhi Ma et al.ACL 2026
  3. SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQLGeonho Lee, Min-Soo KimVLDB 2026
  4. ToolSQL: A Tool-Assisted Agent for SQL Verification and RefinementZhongyuan Wang, Richong Zhang, Zhijie Nie, Jaein KimKDD 2025 · 2 citations
  5. SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQLYue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya et al.NeurIPS 2025 · 21 citations
Dusk over the sea between two cliffs drawn in fine vertical lines