A Comparative Evaluation of Schema Subsetting for LLM-based NL-to-SQL over Large-Schema Databases

Large Language Models (LLMs) have become the standard for natural language interfaces to databases, but their effectiveness can be limited by context window constraints, especially for databases with large schemas. Schema subsetting or linking, which is the task of reducing the schema information provided to the LLM, has emerged as a strategy to address these limitations, yet its impact on NL-to-SQL performance remains unclear, particularly for very large schemas. In this paper, we systematically evaluate 7 real-world schema subsetting modules across 3 contemporary NL-to-SQL benchmarks, including Bird, Spider 2, and SNAILS, and we introduce BigBird—an expansion of the Bird benchmark datasets that provides additional data for evaluating subsetting of large schemas. We also introduce new subsetting-specific performance and efficiency metrics that enable in-depth evaluation of subsetting methods. Our analysis aligns with other recent work that suggests that most subsetting methods actually degrade NL-to-SQL execution accuracy from between 3% -10% (model and method dependent) on smaller schemas, but also reveals that some subsetting methods can improve NL-to-SQL execution accuracy by up to 2% - 7% and others reduce token usage while generally maintaining the same execution accuracy performance as full-schema representations on large schemas. We also present SKALPEL, a prototype hybrid subsetting method that combines LLM-based question decomposition with semantic search, suggesting the potential for reduced token usage in NL-to-SQL workflows. These findings clarify the trade-offs of schema subsetting and motivate future research on scalable schema linking for large databases.