ACL Findings2023
NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing
Tingting Wu, Xiao Ding, Minji Tang, Hao Zhang, Bing Qin, Ting Liu
Abstract
Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade model generalization. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance. Due to the lack of suitable datasets, previous studies have frequently employed synthetic label noise to mimic realworld label noise. However, synthetic noise is not instance-dependent, making this approximation not always effective in practice. Recent research has proposed benchmarks for learning with real-world noisy labels. However, the noise sources within may be single or fuzzy, making benchmarks different from data with heterogeneous label noises in the real world. To tackle these issues, we contribute Noisywik-iHow, the largest NLP benchmark built with minimal supervision. Specifically, inspired by human cognition, we explicitly construct multiple sources of label noise to imitate human errors throughout the annotation, replicating realworld noise, whose corruption is affected by both ground-truth labels and instances. Moreover, we provide a variety of noise levels to support controlled experiments on noisy data, enabling us to evaluate LNL methods systematically and comprehensively. After that, we conduct extensive multi-dimensional experiments on a broad range of LNL methods, obtaining new and intriguing findings. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise cites Does label smoothing mitigate label noise?
- Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise cites Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
- NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition cites Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
Cited by top-tier papers4
- NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity RecognitionElena Merdjanovska, Ansar Aynetdinov, Alan AkbikEMNLP 2024 · 5 citations
- Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label LearningYeachan Kim, Junho Kim, SangKeun LeeACL 2024 · 4 citations
- Fault Lines: Benchmarking the Impact of Label Data Quality on ML Robustness and FairnessDavid Jackson, Paul Groth, Hazar HarmouchVLDB 2026
- Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension abilityYujin Han, Lei Xu, Sirui Chen, Difan Zou et al.ICLR 2025
Builds on10
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Decoupling Representation and Classifier for Long-Tailed RecognitionBingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan et al.ICLR 2020 · 1,496 citations
- Does label smoothing mitigate label noise?Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv KumarICML 2020 · 411 citations
- Beyond Synthetic Noise: Deep Learning on Controlled Noisy LabelsLu Jiang, Di Huang, Mason Liu, Weilong YangICML 2020 · 241 citations
- Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label NoisePengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao et al.AAAI 2021 · 156 citations
Related papers
- Unveiling Open-set Noise: Theoretical Insights into Label NoiseChen Feng, Nicu Sebe, Georgios Tzimiropoulos, Miguel R. D. Rodrigues et al.ACM MM 2025
- NoiseGPT: Label Noise Detection and Rectification through Probability CurvatureHaoyu Wang, Zhuo Huang, Zhiwei Lin, Tongliang LiuNeurIPS 2024 · 27 citations
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic OptimizationKuan Zhang, Chengliang Chai, Jingzhe Xu, Chi Zhang et al.NeurIPS 2025 · 6 citations
- Learning with Structural Labels for Learning with Noisy LabelsNoo-Ri Kim, Jin-Seop Lee, Jee-Hyong LeeCVPR 2024
- Learning to Clean: Reinforcement Learning for Noisy Label CorrectionMarzi Heidari, Hanping Zhang, Yuhong GuoNeurIPS 2025
