BenchIE⌃FL: A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark

Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose BenchIE F L , a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. BenchIE F L allows insightful conclusions to be drawn on the actual performance of OIE extractors. Cluster 6: Marc -has [a] -label Marc -has -[a] label Marc -has -[own] label Marc -owns [a] -label Marc -owns -[a] label Table 21: Annotations examples from BenchIE and BenchIE F L