Lune

EMNLP Findings2025

Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models

Blanca Calvo Figueras, Rodrigo Agerri

Abstract

The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including 5K manually annotated questions. We also investigate automatic evaluation methods and propose reference-based techniques as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code 1 plus a public leaderboard are provided to encourage further research, not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin, Yuning Mao et al.EMNLP 2022 · 103 citations
  2. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkOscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz et al.EMNLP Findings 2023 · 85 citations
  3. Semantic Graphs for Generating Deep QuestionsLiangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua et al.ACL 2020 · 79 citations
  4. BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations
  5. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language ModelsSeungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin et al.EMNLP 2024 · 38 citations

Related papers

  1. ArgBench: Benchmarking LLMs on Computational Argumentation TasksYamen Ajjour, Carlotta Quensel, Nedim Lipka, Henning WachsmuthACL Findings 2026 · 1 citation
  2. Natural Language Reasoning in Large Language Models: Analysis and EvaluationDebela Gemechu, Ramon Ruiz-Dolz, Henrike Beyer, Chris ReedACL Findings 2025 · 6 citations
  3. TuringQ: Benchmarking AI Comprehension in Theory of ComputationPardis Sadat Zahraei, Ehsaneddin AsgariEMNLP Findings 2024 · 1 citation
  4. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding TasksHongchao Jiang, Yiming Chen, Yushi Cao, Hung-Yi Lee et al.ACL 2026 · 33 citations
  5. What Has Been Lost with Synthetic Evaluation?Alexander Gill, Abhilasha Ravichander, Ana MarasovicEMNLP Findings 2025 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines