Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

Recent progress in Text-to-SQL research has led to remarkable performance in static, single-turn settings, where models generate SQL queries from natural language questions through multi-step reasoning. However, such systems remain inadequate for real-world, interactive applications, where user intents evolve dynamically and queries must be refined over multiple turns. In practical scenarios such as financial or business analytics, users iteratively modify query constraints or dimensions based on intermediate results. To systematically evaluate these capabilities, we introduce DySQL-Bench, a benchmark designed to assess model's ability under dynamic user interactions. Unlike previous manually curated datasets, DySQL-Bench is constructed through an automated two-stage pipeline consisting of task synthesis and task verification. In the synthesis stage, raw database tables are transformed into structured tree representations, each capturing the logical relationships within a complete user record. These structured trees serve as the foundation from which Large Language Models (LLMs) generate diverse and realistic evaluation tasks. In the verification stage, an interaction-oriented quality control module filters out erroneous samples, followed by expert validation to ensure data reliability. Through rigorous human evaluation, our automatic synthesis data achieves 100% correctness. We further propose a multi-turn dynamic evaluation framework simulating a realistic interaction environment involving three roles: an LLM-simulated user, the model under evaluation, and an executable database system. The simulated user interacts with the model based on task instructions and intermediate responses, while the model must adapt its reasoning strategy and SQL generation in response to evolving user intents. DySQL-Bench integrates databases from BIRD and Spider 2, spanning 13 domains and comprising 1,072 evaluation tasks. Experimental results demonstrate that even advanced models such as GPT-4o achieve only 58.34% overall accuracy and 23.81% on the Passˆ5 metric, highlighting the significant challenge posed by our benchmark. All code and benchmark data are publicly available at https://github.com/Aurora-slz/Real- World-SQL-Bench.