Lune

EMNLP Findings2025

From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs?

Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao

Abstract

Despite the rapid development of large language models (LLMs), existing benchmark datasets often focus on low-level cognitive tasks, such as factual recall and basic comprehension, while providing limited coverage of higher-level reasoning skills, including analysis, evaluation, and creation. In this work, we systematically assess the cognitive depth of popular LLM benchmarks using Bloom's Taxonomy to evaluate both the cognitive and knowledge dimensions. Our analysis reveals a pronounced imbalance: most datasets concentrate on 'Remembering' and 'Understanding', with metacognitive and creative reasoning largely underrepresented. We also find that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. These findings highlight the importance of future benchmarks incorporating metacognitive evaluations to more accurately assess and enhance model performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CMMLU: Measuring massive multitask language understanding in Chinese cites GLM-130B: An Open Bilingual Pre-trained Model
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
  2. GLM-130B: An Open Bilingual Pre-trained ModelAohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang et al.ICLR 2023 · 295 citations
  3. CMMLU: Measuring massive multitask language understanding in ChineseHaonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang et al.ACL Findings 2024 · 105 citations
  4. Efficient multi-prompt evaluation of LLMsFelipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva et al.NeurIPS 2024 · 93 citations
  5. mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and HumansYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL Findings 2024 · 20 citations

Related papers

  1. Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language ModelsMohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri et al.ACL Findings 2026
  2. Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem SolvingYuxuan Zhou, Xien Liu, Chenwei Yan, Chen Ning et al.ICML 2025
  3. From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice QuestionsDonghyeon Shin, Seungpil Lee, Klea Kovacec, Sundong KimEMNLP Findings 2024 · 1 citation
  4. BIG-Bench Extra HardMehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch et al.ACL 2025
  5. BloomVQA: Assessing Hierarchical Multi-modal ComprehensionYunye Gong, Robik Shrestha, Jared Claypoole, Michael Cogswell et al.ACL Findings 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines