ACL Findings2024
The State of Relation Extraction Data Quality: Is Bigger Always Better?
Erica Cai, Brendan T. O'Connor
Abstract
Relation extraction (RE) extracts structured tuples of relationships (e.g. friend, enemy) between entities (e.g. Sherlock Holmes, John Watson) from text, with exciting potential applications. Hundreds of RE papers have been published in recent years; do their evaluation practices inform these goals? We review recent surveys and a sample of recent RE methods papers, compiling 38 datasets currently being used. Unfortunately, many have frequent label errors, and ones with known problems continue to be used. Many datasets focus on producing labels for a large number of relation types, often through error-prone annotation methods (e.g. distant supervision or crowdsourcing), and many recent papers rely exclusively on such datasets. We draw attention to a promising alternative: datasets with a small number of relations, often in specific domains like chemistry, finance, or biomedicine, where it is possible to obtain high quality expert annotations; such data can more realistically evaluate RE performance. The research community should consider more often using such resources.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on47
- Double Graph Based Reasoning for Document-level Relation ExtractionShuang Zeng, Runxin Xu, Baobao Chang, Lei LiEMNLP 2020 · 238 citations
- GPT-RE: In-context Learning for Relation Extraction using Large Language ModelsZhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu et al.EMNLP 2023 · 132 citations
- Dialogue-Based Relation ExtractionDian Yu, Kai Sun, Claire Cardie, Dong YuACL 2020 · 106 citations
- Label Verbalization and Entailment for Effective Zero and Few-Shot Relation ExtractionOscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena et al.EMNLP 2021 · 94 citations
- SelfORE: Self-supervised Relational Feature Learning for Open Relation ExtractionXuming Hu, Lijie Wen, Yusong Xu, Chenwei Zhang et al.EMNLP 2020 · 81 citations
Related papers
- Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation ExtractionTianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu et al.ACL Findings 2021 · 29 citations
- Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation ExtractionPengshuai Li, Xinsong Zhang, Weijia Jia, Wei ZhaoEMNLP Findings 2020 · 5 citations
- Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocREDQuzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu et al.ACL 2022
- REDFM: a Filtered and Multilingual Relation Extraction DatasetPere-Lluís Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, Roberto NavigliACL 2023 · 9 citations
- CrossRE: A Cross-Domain Dataset for Relation ExtractionElisa Bassignana, Barbara PlankEMNLP Findings 2022 · 15 citations
