EMNLP Findings2023
Evaluating the Knowledge Base Completion Potential of GPT
Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo, Gerhard Weikum
Abstract
Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT's potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li et al.ICLR 2024 · 867 citations
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric MemoriesAlex Mallen, Akari Asai, Victor Zhong, Rajarshi Das et al.ACL 2023 · 233 citations
- CoDEx: A Comprehensive Knowledge Graph Completion BenchmarkTara Safavi, Danai KoutraEMNLP 2020 · 97 citations
- Editing Factual Knowledge in Language ModelsNicola De Cao, Wilker Aziz, Ivan TitovEMNLP 2021 · 20 citations
Related papers
- Better Together: Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood InformationAlla Chepurova, Aydar Bulatov, Yuri Kuratov, Mikhail BurtsevEMNLP Findings 2023 · 9 citations
- Enabling LLM Knowledge Analysis via Extensive MaterializationYujia Hu, Tuan-Phong Nguyen, Shrestha Ghosh, Simon RazniewskiACL 2025
- IELM: An Open Information Extraction Benchmark for Pre-Trained Language ModelsChenguang Wang, Xiao Liu, Dawn SongEMNLP 2022 · 3 citations
- Knowledge Context Modeling with Pre-trained Language Models for Contrastive Knowledge Graph CompletionGuangqian Yang, Yi Liu, Lei Zhang, Licheng Zhang et al.ACL Findings 2024 · 9 citations
- From Existence to Exhaustiveness: Unveiling the Compounding Failures of LLMs in Multi-answer Event Temporal ReasoningShaojuan WuSIGIR 2026
