Lune

EMNLP Findings2023

Knowledge Corpus Error in Question Answering

Yejoon Lee, Philhoon Oh, James Thorne

Abstract

Recent works in open-domain question answering (QA) have explored generating context passages from large language models (LLMs), replacing the traditional retrieval step in the QA pipeline. However, it is not well understood why generated passages can be more effective than retrieved ones. This study revisits the conventional formulation of QA and introduces the concept of knowledge corpus error. This error arises when the knowledge corpus used for retrieval is only a subset of the entire string space, potentially excluding more helpful passages that exist outside the corpus. LLMs may mitigate this shortcoming by generating passages in a larger space. We come up with an experiment of paraphrasing human-annotated gold context using LLMs to observe knowledge corpus error empirically. Our results across three QA benchmarks reveal an increased performance (10% -13%) when using paraphrased passage, indicating a signal for the existence of knowledge corpus error. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Finetuned Language Models are Zero-Shot Learners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  4. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  5. Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations

Related papers

  1. Merging Generated and Retrieved Knowledge for Open-Domain QAYunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee et al.EMNLP 2023 · 12 citations
  2. SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMsJaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park et al.ICLR 2024 · 89 citations
  3. QPaug: Question and Passage Augmentation for Open-Domain Question Answering of LLMsMinsang Kim, Cheoneum Park, Seung BaekEMNLP Findings 2024 · 5 citations
  4. Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kChihiro Taguchi, Seiji Maekawa, Nikita BhutaniEMNLP 2025
  5. Improving Passage Retrieval with Zero-Shot Question GenerationDevendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan et al.EMNLP 2022 · 69 citations
Dusk over the sea between two cliffs drawn in fine vertical lines