Lune

SIGIR2026Top-tier venue

From Existence to Exhaustiveness: Unveiling the Compounding Failures of LLMs in Multi-answer Event Temporal Reasoning

Shaojuan Wu

Abstract

Large Language Models (LLMs) have achieved remarkable success in temporal reasoning. However, existing benchmarks predominantly adopt a "single-answer" paradigm, focusing on verifying the existence of a specific fact while overlooking the challenge of exhaustiveness. In real-world scenarios, entities often simultaneously play multiple roles or exist in multiple states within the same timeframe. To bridge this gap, we introduce MulTR, a comprehensive benchmark designed for multi-answer temporal reasoning from long unstructured contexts. Specifically, MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia through a logic-driven synthesis process. Notably, we formulate two distinct settings, question-dependent and document-dependent, based on the presence of cue words in the question. It is designed to systematically decouple temporal reasoning capabilities from the uncertainty of the number of answers. Experiment results demonstrate that state-of-the-art models suffer from retrieval laziness, terminating the search process prematurely after locating the first valid piece of evidence. Consequently, their performance drops sharply when evaluated on strict exact match metrics. MulTR, as a diagnostic testing platform, reveal these defects and establish the rigorous standard for future research in dynamic knowledge processing. The MulTR benchmark and evaluation prompt are publicly available at https://github.com/TemporalNLP/MulTR.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

its abstract · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language ModelsZheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu et al.ACL 2024 · 12 citations
  2. ComplexTempQA: A 100m Dataset for Complex Temporal Question AnsweringRaphael Gruber, Abdelrahman Abdallah, Michael Färber, Adam JatowtEMNLP 2025 · 2 citations
  3. TRAM: Benchmarking Temporal Reasoning for Large Language ModelsYuqing Wang, Yun ZhaoACL Findings 2024 · 24 citations
  4. ETRQA: A Comprehensive Benchmark for Evaluating Event Temporal Reasoning Abilities of Large Language ModelsSigang Luo, Yinan Liu, Dongying Lin, Yingying Zhai et al.ACL Findings 2025 · 3 citations
  5. TempTabQA: Temporal Question Answering for Semi-Structured TablesVivek Gupta, Pranshu Kandoi, Mahek Bhavesh Vora, Shuo Zhang et al.EMNLP 2023 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines