Lune

ACL Findings2025

Optimizing Reasoning for Text-to-SQL with Execution Feedback

Bohan Zhai, Canwen Xu, Yuxiong He, Zhewei Yao

Abstract

Text-to-SQL demands precise reasoning to convert natural language questions into structured queries. While large language models (LLMs) excel in many reasoning tasks, their ability to leverage Chain-of-Thought (CoT) reasoning for text-to-SQL remains underexplored. We identify critical limitations: zero-shot CoT offers minimal gains, and Direct Preference Optimization (DPO) applied without CoT yields marginal improvements. We propose ExCoT-DPO, a novel framework that iteratively optimizes open-source LLMs by combining CoT reasoning with off-policy and on-policy DPO, relying solely on execution accuracy as feedback. This approach eliminates the need for reward models or human-annotated preferences. Our experimental results demonstrate significant performance gains: ExCoT-DPO improves execution accuracy on BIRD from 57.37% to 68.51% and on Spider from 78.81% to 86.59% for LLaMA-3 70B, with Qwen-2.5-Coder demonstrating similar improvements. Our best model achieves state-of-the-art performance in the single-model setting on both BIRD and Spider datasets, notably achieving 68.53% on the BIRD test set. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
  • OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
  • OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation cites DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate SupervisionRuilin Hu, Yuyu Luo, Guoliang Li, Shuangqiao Wu et al.VLDB 2026 · 4 citations
  2. stratum: A System Infrastructure for Massive Agent-Centric ML WorkloadsArnab Phani, Elias Strauss, Sebastian SchelterVLDB 2026

Builds on11

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
  4. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
  5. Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationDawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun et al.VLDB 2024 · 609 citations

Related papers

  1. ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQLYaxun Dai, Wenxuan Xie, Xialie Zhuang, Tianyu Yang et al.ACL 2026 · 8 citations
  2. Exploring Chain of Thought Style Prompting for Text-to-SQLChang-Yu Tai, Ziru Chen, Tianshu Zhang, Xiang Deng et al.EMNLP 2023 · 35 citations
  3. Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLHanbing Liu, Haoyang Li, Xiaokang Zhang, Ruotong Chen et al.ACL 2025 · 10 citations
  4. ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-ThoughtHanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu et al.EMNLP Findings 2023 · 54 citations
  5. VET: Verifiable Execution Tracing for Reliable Text-to-SQL GenerationDongyu Wang, Jingyu Li, Lan Zhang, Ganggang Yu et al.ACL Findings 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines