An End-To-End Re-Evaluation of Table Entity-Linkers

Abstract Knowledge graph (KG) entity linkers link entity mentions from a source data representation to their corre sponding entities in a target KG. A knowledge graph (KG) is a popular graph model expressing semantic information about entities, concepts, and relationships. Consequently, entity linking from tables to KGs has become increasingly important, allowing semantic table augmentation and other advanced data integration tasks. However, existing evaluations of entity linkers are incomplete, as they only evaluate for specific applications and only consider aggregated output quality metrics: they do not consider the performance and effectiveness of the individual entity linking components along with their scalability. To address this gap, we provide an in-depth analysis and taxonomy of state of-the-art entity linkers by thoroughly evaluating the quality and scalability of the individual entity linking components. Hence, we evaluate entity linkers on four existing entity linking benchmarks using DBpedia and Wikidata and identify major bottlenecks to be overcome to make these entity linkers fully applicable in real-world use-cases. We identified candidate generation as the most crucial entity linking step, which commonly is overlooked. Furthermore, we show that most entity linkers are irreproducible, either because they are not open-source, or because they use irreproducible public endpoints and datasets. Acknowledgements This research was partially funded by the Danish Council for Independent Research (DFF), grant agreement no. DFF 804800051B, the Poul Due Jensen Fond, and DataGEMS, funded by the European Union’s Horizon Europe Research and Innovation programme, grant agreement no. 101188416.