ACL Findings2025
Iterative Repair with Weak Verifiers for Few-shot Transfer in KBQA with Unanswerability
Riya Sawhney, Samrat Yadav, Indrajit Bhattacharya, Mausam
Abstract
Real-world applications of KBQA require models to detect different types of unanswerable questions with a limited volume of in-domain labeled training data. We propose the novel task of few-shot transfer for KBQA with unanswerable questions. The state-of-the-art KBQA few-shot transfer model (FuSIC-KBQA) uses an iterative repair strategy that assumes that all questions are answerable. As a remedy, we present FUn-FuSIC -a novel solution for our task that extends FuSIC-KBQA with Feedback for Unanswerability (FUn), which is an iterative repair strategy for answerable as well as unanswerable questions. FUn uses feedback from a suite of strong and weak verifiers, and an adaptation of self-consistency for unanswerability for assessing answerability of questions. Our experiments show that FUn-FuSIC significantly outperforms suitable adaptations of multiple LLM-based and supervised SoTA models on our task, while establishing a new SoTA performance for answerable few-shot transfer as well. We have made datasets and other resources publicly available 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Teaching Large Language Models to Self-Debug cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Teaching Large Language Models to Self-Debug cites Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: language agents with verbal reinforcement learning cites Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: language agents with verbal reinforcement learning cites Teaching Large Language Models to Self-Debug
Builds on20
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Teaching Large Language Models to Self-DebugXinyun Chen, Maxwell Lin, Nathanael Schärli, Denny ZhouICLR 2024 · 1,085 citations
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
Related papers
- Few-shot Transfer Learning for Knowledge Base Question Answering: Fusing Supervised Models with In-Context LearningMayur Patidar, Riya Sawhney, Avinash Kumar Singh, Biswajit Chatterjee et al.ACL 2024
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question AnsweringZhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li et al.AAAI 2024 · 143 citations
- RetinaQA: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable QuestionsPrayushi Faldu, Indrajit Bhattacharya, MausamACL 2024 · 2 citations
- Harvesting and Refining Question-Answer Pairs for Unsupervised QAZhongli Li, Wenhui Wang, Li Dong, Furu Wei et al.ACL 2020 · 29 citations
- Few-shot In-context Learning on Knowledge Base Question AnsweringTianle Li, Xueguang Ma, Alex Zhuang, Yu Gu et al.ACL 2023 · 55 citations
