Lune

EMNLP Findings2023

Gradually Excavating External Knowledge for Implicit Complex Question Answering

Chang Liu, Xiaoguang Li, Lifeng Shang, Xin Jiang, Qun Liu, Edmund Y. Lam, Ngai Wong

Abstract

Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA for ∼10B-scale LLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Evaluation Guided Beam Search for Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions cites Dense Passage Retrieval for Open-Domain Question Answering
  • Is ChatGPT a General-Purpose Natural Language Processing Task Solver? cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  2. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen et al.EMNLP 2023 · 449 citations
  3. Self-Evaluation Guided Beam Search for ReasoningYuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao et al.NeurIPS 2023 · 316 citations
  4. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step QuestionsHarsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish SabharwalACL 2023 · 187 citations
  5. Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations

Related papers

  1. Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question AnsweringHongda Sun, Yuxuan Liu, Chengwei Wu, Haiyu Yan et al.WWW 2024 · 16 citations
  2. ProgRAG: Hallucination-Resistant Progressive Retrieval and Reasoning over Knowledge GraphsMinbae Park, Hyemin Yang, Jeonghyun Kim, Kunsoo Park et al.AAAI 2026
  3. Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language ModelsYifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen et al.ACL Findings 2024 · 31 citations
  4. Explore What LLM Does Not Know in Complex Question AnsweringXin Lin, Zhenya Huang, Zhiqiang Zhang, Jun Zhou et al.AAAI 2025 · 8 citations
  5. Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex QuestionsShulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv et al.EMNLP Findings 2023 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines