Lune

EMNLP Findings2025

Token-Level Metrics for Detecting Incorrect Gold Annotations in Named Entity Recognition

Elena Merdjanovska, Alan Akbik

Abstract

Annotated datasets for supervised learning tasks often contain incorrect gold annotations, i.e. label noise. To address this issue, many noisy label learning approaches incorporate metrics to filter out unreliable samples, for example using heuristics such as high loss or low confidence. However, when these metrics are integrated into larger pipelines, it becomes difficult to compare their effectiveness, and understand their individual contribution to reducing label noise. This paper directly compares popular sample metrics for detecting incorrect annotations in named entity recognition (NER). NER is commonly approached as token classification, so the metrics are calculated for each training token and the incorrect ones are flagged by defining metrics thresholds. We compare the metrics based on (i) their accuracy in detecting the incorrect labels and (ii) the test scores when retraining a model using the cleaned dataset. We show that training dynamics metrics work the best overall. The best metrics effectively reduce the label noise across different noise types. The errors that the model has not yet memorized are more feasible to detect, and relabeling these tokens is a more effective strategy than excluding them from training. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Identifying Mislabeled Data using the Area Under the Margin RankingGeoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, Kilian Q. WeinbergerNeurIPS 2020 · 398 citations
  2. Deep Learning Through the Lens of Example DifficultyRobert J. N. Baldock, Hartmut Maennel, Behnam NeyshaburNeurIPS 2021 · 204 citations
  3. Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label NoisePengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao et al.AAAI 2021 · 156 citations
  4. BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant SupervisionChen Liang, Yue Yu, Haoming Jiang, Siawpeng Er et al.KDD 2020 · 118 citations
  5. Learning from Noisy Labels for Entity-Centric Information ExtractionWenxuan Zhou, Muhao ChenEMNLP 2021 · 32 citations

Related papers

  1. NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity RecognitionElena Merdjanovska, Ansar Aynetdinov, Alan AkbikEMNLP 2024 · 5 citations
  2. Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity RecognitionZhiyuan Ma, Jintao Du, Shuheng ZhouACL Findings 2023 · 4 citations
  3. CleanCoNLL: A Nearly Noise-Free Named Entity Recognition DatasetSusanna Rücker, Alan AkbikEMNLP 2023 · 3 citations
  4. LANE: Label-Aware Noise Elimination for Fine-Grained Text ClassificationTiberiu Sosea, Cornelia CarageaICLR 2026
  5. Learning Discriminative Dynamics with Label Corruption for Noisy Label DetectionSuyeon Kim, Dongha Lee, SeongKu Kang, Sukang Chae et al.CVPR 2024
Dusk over the sea between two cliffs drawn in fine vertical lines