Lune

EMNLP Findings2024

Rater Cohesion and Quality from a Vicarious Perspective

Deepak Pandita, Tharindu Cyril Weerasooriya, Sujan Dutta, Sarah Luger, Tharindu Ranasinghe, Ashiqur R. KhudaBukhsh, Marcos Zampieri, Christopher Homan

Abstract

This paper discusses and contains offensive content. Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater-cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater-quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation cites Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
  • Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech cites Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation
  • Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech cites Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope SpeechJonathan Pofcher, Christopher M. Homan, Randall Sell, Ashiqur R. KhudaBukhshEMNLP 2025

Builds on6

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Toward a Perspectivist Turn in Ground Truthing for Predictive ComputingFederico Cabitza, Andrea Campagner, Valerio BasileAAAI 2023 · 236 citations
  3. Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity AnnotationNitesh Goyal, Ian D. Kivlichan, Rachel Rosen, Lucy VassermanCSCW 2022 · 74 citations
  4. From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language ModelsJulia Mendelsohn, Ronan Le Bras, Yejin Choi, Maarten SapACL 2023 · 14 citations
  5. Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is OffensiveTharindu Cyril Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri et al.EMNLP 2023 · 12 citations

Related papers

  1. Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level LearningTharindu Cyril Weerasooriya, Sarah Luger, Saloni Poddar, Ashiqur R. KhudaBukhsh et al.ACL 2023 · 2 citations
  2. When the Majority is Wrong: Modeling Annotator Disagreement for Subjective TasksEve Fleisig, Rediet Abebe, Dan KleinEMNLP 2023 · 11 citations
  3. PerspectiveMod: A Perspectivist Resource for Deliberative ModerationEva Maria Vecchi, Neele Falk, Carlotta Quensel, Iman Jundi et al.EMNLP 2025
  4. PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI HarmJingjing Li, Joel Mire, Eve Fleisig, Valentina Pyatkin et al.ICLR 2026 · 6 citations
  5. ARTICLE: Annotator Reliability Through In-Context LearningSujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri et al.AAAI 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines