ACL Findings2021
Revisiting the Evaluation of End-to-end Event Extraction
Abstract
Event extraction (EE) aims to harvest event instances from plain text, where each instance is composed of a group of event arguments with specific event roles. Existing end-to-end EE research usually adopts the role-averaged evaluation that produces evaluation measures by averaging evaluation statistics of each event role. However, although this averaged metric can indicate the model performance to some extent, we find that such metric can be pretty misleading to downstream applications that utilize an event instance as a whole, where one wrongly identified event argument can substantially alter the whole meaning of an event instance. To mitigate this gap and provide a more complete understanding of performance, we propose two new evaluation metrics that also consider an event instance as a whole and explicitly penalize wrongly identified event arguments. Moreover, to support diverse preferences of evaluation metrics motivated by different scenarios, we propose a new training paradigm based on reinforcement learning for a typical end-to-end EE model, i.e., Doc2EDAG. Our extensive experiments show that the new training improves the initial one by a large margin (about 10%) under new metrics. Nevertheless, the current performance is still far from satisfactory, and optimizing towards these new metrics calls for more future research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Related papers
- Event Extraction by Answering (Almost) Natural QuestionsXinya Du, Claire CardieEMNLP 2020 · 391 citations
- Document-level Event Extraction via Parallel Prediction NetworksHang Yang, Dianbo Sui, Yubo Chen, Kang Liu et al.ACL 2021
- HTMR: Hybrid Token Masking Reinforcement Learning with Verifiable Rewards for Event Argument Extraction with Multi-Perspective ReasoningJianwen Luo, Yongkang Jin, Yu Hong, Jianmin YaoACL 2026
- Contextualized Soft Prompts for Extraction of Event ArgumentsChien Van Nguyen, Hieu Man, Thien Huu NguyenACL Findings 2023 · 20 citations
- Boosting Event Extraction with Denoised Structure-to-Text AugmentationBo Wang, Heyan Huang, Xiaochi Wei, Ge Shi et al.ACL Findings 2023 · 6 citations
