Lune

EMNLP Findings2025

REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction

Omar Sharif, Joseph Gatto, Madhusudan Basak, Sarah Masud Preum

Abstract

Event argument extraction identifies arguments for predefined event roles in text. Existing work evaluates this task with exact match (EM), where predicted arguments must align exactly with annotated spans. While suitable for spanbased models, this approach falls short for large language models (LLMs), which often generate diverse yet semantically accurate arguments. EM severely underestimates performance by disregarding valid variations. Furthermore, EM evaluation fails to capture implicit arguments (unstated but inferable) and scattered arguments (distributed across a document). These limitations underscore the need for an evaluation framework that better captures models' actual performance.

To bridge this gap, we introduce REGen, a Reliable Evaluation framework for Generative event argument extraction. REGen combines the strengths of exact, relaxed, and LLM-based matching to better align with human judgment. Experiments on six datasets show that REGen reveals an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. Human validation further confirms REGen's effectiveness, achieving 87.67% alignment with human assessments of argument correctness.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learning to Generate and Extract: A Multi-Agent Collaboration Framework for Zero-Shot Document-Level Event Arguments Extraction cites Event Extraction by Answering (Almost) Natural Questions
  • Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Learning to Generate and Extract: A Multi-Agent Collaboration Framework for Zero-Shot Document-Level Event Arguments ExtractionGuangjun Zhang, Hu Zhang, Yazhou Han, Yue Fan et al.AAAI 2026

Builds on12

  1. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM PromptsJ. D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, Qian YangCHI 2023 · 892 citations
  2. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  3. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
  4. Event Extraction by Answering (Almost) Natural QuestionsXinya Du, Claire CardieEMNLP 2020 · 391 citations
  5. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language ModelsLei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu et al.ACL 2023 · 249 citations

Related papers

  1. Thinking about how to extract: Energizing LLMs' emergence capabilities for document-level event argument extractionKai Shuang, Ji Zhou, Qiwei Wang, Jinyu GuoACL Findings 2024 · 6 citations
  2. Revisiting the Evaluation of End-to-end Event ExtractionShun Zheng, Wei Cao, Wei Xu, Jiang BianACL Findings 2021 · 4 citations
  3. REAR: Reinforced Reasoning Optimization for Event Argument Extraction with Relation-Aware SupportJianwen Luo, Yu Hong, Shuai Yang, Jianmin YaoEMNLP Findings 2025
  4. Explicit Role Interaction Network for Event Argument ExtractionNan Ding, Chunming Hu, Kai Sun, Samuel Mensah et al.EMNLP Findings 2022 · 6 citations
  5. xFinder: Large Language Models as Automated Evaluators for Reliable EvaluationQingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines