EMNLP Findings2020
OptSLA: an Optimization-Based Approach for Sequential Label Aggregation
Nasim Sabetpour, Adithya Kulkarni, Qi Li
Abstract
The need for the annotated training dataset on which data-hungry machine learning algorithms feed has increased dramatically with advanced acclaim of machine learning applications. To annotate the data, people with domain expertise are needed, but they are seldom available and expensive to hire. This has lead to the thriving of crowdsourcing platforms such as Amazon Mechanical Turk (AMT). However, the annotations provided by one worker cannot be used directly to train the model due to the lack of expertise. Existing literature in annotation aggregation focuses on binary and multi-choice problems. In contrast, little work has been done on complex tasks such as sequence labeling with imbalanced classes, a ubiquitous task in Natural Language Processing (NLP), and Bio-Informatics. We propose OPTSLA, an Optimization-based Sequential Label Aggregation method, that jointly considers the characteristics of sequential labeling tasks, workers reliabilities, and advanced deep learning techniques to conquer the challenge. We evaluate our model on crowdsourced data for named entity recognition task. Our results show that the proposed OPTSLA outperforms the state-of-the-art aggregation methods, and the results are easier to interpret.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Related papers
- Low Resource Sequence Tagging with Weak LabelsEdwin Simpson, Jonas Pfeiffer, Iryna GurevychAAAI 2020 · 14 citations
- Semi-Supervised Multi-Label Learning from Crowds via Deep Sequential Generative ModelWanli Shi, Victor S. Sheng, Xiang Li, Bin GuKDD 2020 · 12 citations
- Online Label Aggregation: A Variational Bayesian ApproachChi Hong, Amirmasoud Ghiassi, Yichi Zhou, Robert Birke et al.WWW 2021 · 8 citations
- Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity RecognitionXin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang et al.ACL 2021
- The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical TurkBenjamin V. Hanrahan, Anita Chen, Jiahua Ma, Ning F. Ma et al.CSCW 2021 · 16 citations
