Lune

EMNLP Findings2025

Time to Revisit Exact Match

Auss Abbood, Zaiqiao Meng, Nigel Collier

Abstract

Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAn-swerQA, a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both 20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models' understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models' most frequent error is to deviate by only ±1 from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings 1 underscore the need for specialised metrics for temporal QA tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TRAM: Benchmarking Temporal Reasoning for Large Language Models cites Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
  • TRAM: Benchmarking Temporal Reasoning for Large Language Models cites MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
  • A Study into Investigating Temporal Robustness of LLMs cites TRAM: Benchmarking Temporal Reasoning for Large Language Models
  • MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models cites Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. TRAM: Benchmarking Temporal Reasoning for Large Language ModelsYuqing Wang, Yun ZhaoACL Findings 2024 · 24 citations
  3. Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language ModelsQingyu Tan, Hwee Tou Ng, Lidong BingACL 2023 · 24 citations
  4. A Study into Investigating Temporal Robustness of LLMsJonas Wallat, Abdelrahman Abdallah, Adam Jatowt, Avishek AnandACL Findings 2025 · 19 citations
  5. MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language ModelsYifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu et al.EMNLP Findings 2023 · 14 citations

Related papers

  1. UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' MemorizationMd Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth et al.ACL 2025 · 18 citations
  2. Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in LLMsSoyeon Kim, Jindong Wang, Xing Xie, Steven Euijong WhangICLR 2026
  3. Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational UnderstandingXinying Qian, Ying Zhang, Xuhui Sui, Yu Zhao et al.ACL 2026
  4. It's High Time: A Survey of Temporal Question AnsweringBhawna Piryani, Abdelrahman Abdallah, Jamshid Mozafari, Avishek Anand et al.ACL 2026 · 6 citations
  5. Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu et al.ACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines