EMNLP Findings2024
SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions
Shicheng Liu, Sina J. Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, Monica S. Lam
Abstract
Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas.
We introduce the SPINACH dataset, an expertannotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset.
We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in F 1 , respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4based KBQA agent, by at least 38.1% in F 1 . 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LLM-MatLogic: Executable Exchange Contracts for Knowledge-Graph Query Answering with Scoped Negation cites Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
- CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era cites Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
- CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era cites Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
- Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphs cites Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
- Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphs cites Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
- Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases cites Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Cited by top-tier papers8
- Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge GraphsJia Ao Sun, Hao Yu, Fabrizio Gotti, Fengran Mo et al.KDD 2026 · 8 citations
- SCRIBES: Web-Scale Script-Based Semi-Structured Data Extraction with Reinforcement LearningShicheng Liu, Kai Sun, Lisheng Fu, Xilun Chen et al.ICLR 2026 · 2 citations
- Controllable and Reliable Knowledge-Intensive Task-Oriented Conversational Agents with Declarative Genie WorksheetsHarshit Joshi, Shicheng Liu, James Chen, Larsen Weigle et al.ACL 2025
- LLM-MatLogic: Executable Exchange Contracts for Knowledge-Graph Query Answering with Scoped NegationDezhuang Miao, Xiaoming Zhang, Bo Zhang, Yibin Du et al.ICML 2026
- CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM EraYanlin Feng, Simone Papicchio, Sajjadur RahmanACL 2025
Builds on17
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Reasoning on Graphs: Faithful and Interpretable Large Language Model ReasoningLinhao Luo, Yuan-Fang Li, Gholamreza Haffari, Shirui PanICLR 2024 · 499 citations
- Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman et al.ICLR 2020 · 401 citations
- Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge BasesYu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler et al.WWW 2021 · 304 citations
Related papers
- Beyond Seen Data: Improving KBQA Generalization Through Schema-Guided Logical Form GenerationShengxiang Gao, Jey Han Lau, Jianzhong QiEMNLP 2025
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question AnsweringZhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li et al.AAAI 2024 · 143 citations
- KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge BaseShulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie et al.ACL 2022
- KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question AnsweringXin Sun, Zhongqi Chen, Xing Zheng, Bowen Song et al.ICML 2026 · 2 citations
- KaeDe: Progressive Generation of Logical Forms via Knowledge-Aware Question Decomposition for Improved KBQARanran Bu, Jian Cao, Jianqi Gao, Shiyou Qian et al.EMNLP Findings 2025
