ACL Findings2025
Optimizing Reasoning for Text-to-SQL with Execution Feedback
Abstract
Text-to-SQL demands precise reasoning to convert natural language questions into structured queries. While large language models (LLMs) excel in many reasoning tasks, their ability to leverage Chain-of-Thought (CoT) reasoning for text-to-SQL remains underexplored. We identify critical limitations: zero-shot CoT offers minimal gains, and Direct Preference Optimization (DPO) applied without CoT yields marginal improvements. We propose ExCoT-DPO, a novel framework that iteratively optimizes open-source LLMs by combining CoT reasoning with off-policy and on-policy DPO, relying solely on execution accuracy as feedback. This approach eliminates the need for reward models or human-annotated preferences. Our experimental results demonstrate significant performance gains: ExCoT-DPO improves execution accuracy on BIRD from 57.37% to 68.51% and on Spider from 78.81% to 86.59% for LLaMA-3 70B, with Qwen-2.5-Coder demonstrating similar improvements. Our best model achieves state-of-the-art performance in the single-model setting on both BIRD and Spider datasets, notably achieving 68.53% on the BIRD test set. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
Cited by top-tier papers2
- OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate SupervisionRuilin Hu, Yuyu Luo, Guoliang Li, Shuangqiao Wu et al.VLDB 2026 · 4 citations
- stratum: A System Infrastructure for Massive Agent-Centric ML WorkloadsArnab Phani, Elias Strauss, Sebastian SchelterVLDB 2026
Builds on11
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
- Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationDawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun et al.VLDB 2024 · 609 citations
Related papers
- ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQLYaxun Dai, Wenxuan Xie, Xialie Zhuang, Tianyu Yang et al.ACL 2026 · 8 citations
- Exploring Chain of Thought Style Prompting for Text-to-SQLChang-Yu Tai, Ziru Chen, Tianshu Zhang, Xiang Deng et al.EMNLP 2023 · 35 citations
- Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLHanbing Liu, Haoyang Li, Xiaokang Zhang, Ruotong Chen et al.ACL 2025 · 10 citations
- ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-ThoughtHanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu et al.EMNLP Findings 2023 · 54 citations
- VET: Verifiable Execution Tracing for Reliable Text-to-SQL GenerationDongyu Wang, Jingyu Li, Lan Zhang, Ganggang Yu et al.ACL Findings 2026 · 1 citation
