HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
Abstract
A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-3.5-turbo. We propose a cooperative game dubbed "HiddenTables" as a potential resolution to this challenge. In essence, "HiddenTables" is played between the code-generating LLM "Solver" and the "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of "HiddenTables" to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns & labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring and Narrowing the Compositionality Gap in Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Binding Language Models in Symbolic Languages cites TAPEX: Table Pre-training via Learning a Neural SQL Executor
- Binding Language Models in Symbolic Languages cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Is There No Such Thing as a Bad Question? H4R: HalluciBot for Ratiocination, Rewriting, Ranking, and Routing cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers1
Builds on7
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- TAPEX: Table Pre-training via Learning a Neural SQL ExecutorQian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi et al.ICLR 2022 · 347 citations
- Measuring and Narrowing the Compositionality Gap in Language ModelsOfir Press, Muru Zhang, Sewon Min, Ludwig Schmidt et al.EMNLP Findings 2023 · 296 citations
- Binding Language Models in Symbolic LanguagesZhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li et al.ICLR 2023 · 38 citations
Related papers
- Large Language Models are Complex Table ParsersBowen Zhao, Changkai Ji, Yuejie Zhang, Wen He et al.EMNLP 2023 · 17 citations
- Table-GPT: Table Fine-tuned GPT for Diverse Table TasksPeng Li, Yeye He, Dror Yashar, Weiwei Cui et al.SIGMOD 2024 · 63 citations
- HCT-QA: A Benchmark for Question Answering on Human-Centric TablesMohammad Shahmeer Ahmad, Zan Ahmad Naeem, Michaël Aupetit, Ahmed K. Elmagarmid et al.ICDE 2026
- Uncovering Limitations of Large Language Models in Information Seeking from TablesChaoxu Pang, Yixuan Cao, Chunhao Yang, Ping LuoACL Findings 2024 · 6 citations
- Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Fine-tuningJunjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong et al.EMNLP 2025 · 2 citations
