Lune

EMNLP Findings2022

Can Language Models Serve as Temporal Knowledge Bases?

Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang, Hai Jin

Abstract

Recent progress regarding the use of language models (LMs) as knowledge bases (KBs) has shown that language models can act as structured knowledge bases for storing relational facts. However, most existing works only considered the LM-as-KB paradigm in a static setting, which ignores the analysis of temporal dynamics of world knowledge. Furthermore, a basic function of KBs, i.e., the ability to store conflicting information (i.e., 1-N, N-1 and N-M relations), is underexplored. In this paper, we formulate two practical requirements for treating LMs as temporal KBs: (i) the capacity to store temporally-scoped knowledge that contains conflicting information and (ii) the ability to use stored knowledge for temporally-scoped knowledge queries. We introduce a new dataset called LAMA-TK which is aimed at probing temporally-scoped knowledge, and investigate the two above requirements to explore the LM-as-KB paradigm in the temporal domain. On the one hand, experiments show that LMs can memorize millions of temporally-scoped facts with relatively high accuracy and transfer stored knowledge to temporal knowledge queries, thereby expanding the LM-as-KB paradigm to the temporal domain. On the other hand, we show that memorizing conflicting information, which has been neglected by previous works, is still challenging for LMs and hinders the memorization of other unrelated one-to-one relationships.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Editing Factual Knowledge in Language Models cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Editing Factual Knowledge in Language Models cites Entities as Experts: Sparse Memory Access with Entity Supervision
  • Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Multi-Stage Prompting for Knowledgeable Dialogue Generation cites Evaluating Commonsense in Pre-Trained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language ModelsYuyang Bai, Shangbin Feng, Vidhisha Balachandran, Zhaoxuan Tan et al.WWW 2024 · 6 citations
  2. Once Upon a Time in Graph: Relative-Time Pretraining for Complex Temporal ReasoningSen Yang, Xin Li, Lidong Bing, Wai LamEMNLP 2023 · 1 citation
  3. Back to the Future: Towards Explainable Temporal Reasoning with Large Language ModelsChenhan Yuan, Qianqian Xie, Jimin Huang, Sophia AnaniadouWWW 2024

Builds on8

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. Evaluating Commonsense in Pre-Trained Language ModelsXuhui Zhou, Yue Zhang, Leyang Cui, Dandan HuangAAAI 2020 · 198 citations
  3. Multi-Stage Prompting for Knowledgeable Dialogue GenerationZihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye et al.ACL Findings 2022 · 45 citations
  4. Entities as Experts: Sparse Memory Access with Entity SupervisionThibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi et al.EMNLP 2020 · 39 citations
  5. Editing Factual Knowledge in Language ModelsNicola De Cao, Wilker Aziz, Ivan TitovEMNLP 2021 · 20 citations

Related papers

  1. DYNAMICQA: Tracing Internal Knowledge Conflicts in Language ModelsSara Marjanovic, Haeun Yu, Pepa Atanasova, Maria Maistro et al.EMNLP Findings 2024 · 5 citations
  2. Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language ModelsPaul Youssef, Osman Alperen Koras, Meijie Li, Jörg Schlötterer et al.EMNLP Findings 2023 · 9 citations
  3. FictionalQA: A Dataset for Studying Memorization and Knowledge AcquisitionJohn Kirchenbauer, Natjanan Mongkolsupawan, Yuxin Wen, Tom Goldstein et al.ICLR 2026 · 1 citation
  4. Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language ModelsHimanshu Beniwal, Dishant Patel, Kowsik Nandagopan D, Hritik Ladia et al.EMNLP Findings 2024 · 2 citations
  5. Language Model Analysis for Ontology Subsumption InferenceYuan He, Jiaoyan Chen, Ernesto Jiménez-Ruiz, Hang Dong et al.ACL Findings 2023 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines