Lune

EMNLP Findings2020

A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels

Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, Walter S. Lasecki

Abstract

Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. Inducing Semantic Roles Without SyntaxJulian Michael, Luke ZettlemoyerACL Findings 2021 · 4 citations
  2. Towards Professional Level Crowd Annotation of Expert Domain DataPei Wang, Nuno VasconcelosCVPR 2023
  3. CLAR: A Cross-Lingual Argument Regularizer for Semantic Role LabelingIshan Jindal, Yunyao Li, Siddhartha Brahma, Huaiyu ZhuEMNLP Findings 2020 · 2 citations
  4. Alignment-free Cross-lingual Semantic Role LabelingRui Cai, Mirella LapataEMNLP 2020 · 9 citations
  5. Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument StructuresSimone Conia, Edoardo Barba, Alessandro Scirè, Roberto NavigliEMNLP Findings 2022 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines