Lune

EMNLP Findings2023

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, Kang Liu

Abstract

Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors. Context: [1] Twitter was created by Jack Dorsey, Noah Glass, Biz Stone, and Evan Williams in March 2006 and launched in July of that year. [2] On October 16, 2008, Evan Williams became the CEO, and Dorsey became the chairman of the company. [3] Jack Dorsey rejoined Twitter in March 2011 as Executive Chief of Product Development. [4] In June 2020, Twitter announced that Patrick Pichette would succeed Omid Kordestani as chairman. [5] In November 2021, Jack Dorsey stepped down as CEO and was replaced by Parag Agrawal, the chief technology officer. [6] On October 27, 2022, business magnate Elon Musk acquired Twitter for US$44 billion, gaining control of the platform. [7] On May 12, 2023, Musk announced that he will resign as CEO of Twitter in approximately six weeks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

15 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • It's High Time: A Survey of Temporal Question Answering cites TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
  • It's High Time: A Survey of Temporal Question Answering cites StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
  • GLM-130B: An Open Bilingual Pre-trained Model cites Language Models are Few-Shot Learners
  • Context-faithful Prompting for Large Language Models cites Language Models are Few-Shot Learners
  • Context-faithful Prompting for Large Language Models cites StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
  • StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers15

  1. REMem: Reasoning with Episodic Memory in Language AgentYiheng Shu, Padmaja Jonnalagedda, Xiang Gao, Bernal Jimenez Gutierrez et al.ICLR 2026 · 20 citations
  2. TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language ModelsZheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu et al.ACL 2024 · 12 citations
  3. EvolveBench: A Comprehensive Benchmark for Assessing Temporal Awareness in LLMs on Evolving KnowledgeZhiyuan Zhu, Yusheng Liao, Zhe Chen, Yuhao Wang et al.ACL 2025 · 10 citations
  4. It's High Time: A Survey of Temporal Question AnsweringBhawna Piryani, Abdelrahman Abdallah, Jamshid Mozafari, Avishek Anand et al.ACL 2026 · 6 citations
  5. Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and OpportunitiesChuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan et al.EMNLP 2025 · 6 citations

Builds on13

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  3. GLM-130B: An Open Bilingual Pre-trained ModelAohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang et al.ICLR 2023 · 295 citations
  4. StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering ModelsAdam Liska, Tomás Kociský, Elena Gribovskaya, Tayfun Terzi et al.ICML 2022 · 129 citations
  5. Context-faithful Prompting for Large Language ModelsWenxuan Zhou, Sheng Zhang, Hoifung Poon, Muhao ChenEMNLP Findings 2023 · 46 citations

Related papers

  1. A Study into Investigating Temporal Robustness of LLMsJonas Wallat, Abdelrahman Abdallah, Adam Jatowt, Avishek AnandACL Findings 2025 · 19 citations
  2. Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language ModelsHimanshu Beniwal, Dishant Patel, Kowsik Nandagopan D, Hritik Ladia et al.EMNLP Findings 2024 · 2 citations
  3. UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' MemorizationMd Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth et al.ACL 2025 · 18 citations
  4. TRAM: Benchmarking Temporal Reasoning for Large Language ModelsYuqing Wang, Yun ZhaoACL Findings 2024 · 24 citations
  5. CRAB: Assessing the Strength of Causal Relationships Between Real-world EventsAngelika Romanou, Syrielle Montariol, Debjit Paul, Léo Laugier et al.EMNLP 2023 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines