Lune

ACL Findings2025

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement

Junyu Lu, Kai Ma, Kaichun Wang, Kelaiti Xiao, Roy Ka-Wei Lee, Bo Xu, Liang Yang, Hongfei Lin

Abstract

Large Language Models (LLMs) have become essential for offensive language detection, yet their ability to handle annotation disagreement remains underexplored. Disagreement samples, which arise from subjective interpretations, pose a unique challenge due to their ambiguous nature. Understanding how LLMs process these cases, particularly their confidence levels, can offer insight into their alignment with human annotators. This study systematically evaluates the performance of multiple LLMs in detecting offensive language at varying levels of annotation agreement. We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making during few-shot learning and instruction finetuning. Our findings reveal that LLMs struggle with low-agreement samples, often exhibiting overconfidence in these ambiguous cases. However, utilizing disagreement samples in training improves both detection accuracy and model alignment with human judgment. These insights provide a foundation for enhancing LLM-based offensive language detection in real-world moderation tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Large Language Models are Zero-Shot Reasoners
  • Large Language Models are Zero-Shot Reasoners cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Probing LLMs for hate speech detection: strengths and vulnerabilities cites Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
  • Unveiling the Implicit Toxicity in Large Language Models cites Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative MontageJinwei Hu, Xinmiao Huang, Youcheng Sun, Yi Dong et al.ACL 2026 · 11 citations
  2. When Generative AI Is Intimate, Sexy, and Violent: Examining Not-Safe-For-Work (NSFW) Chatbots on FlowGPTXian Li, Yuanning Han, Di Liu, Pengcheng An et al.CHI 2026 · 1 citation
  3. Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling using GazeÖzge Alaçam, Sanne Hoeken, Andreas Säuberli, Hannes Gröner et al.EMNLP 2025
  4. Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDBryan Chen Zhengyu Tan, Daniel Wai Kit Chin, Zhengyuan Liu, Nancy F. Chen et al.EMNLP 2025

Builds on10

  1. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  2. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  3. Latent Hatred: A Benchmark for Understanding Implicit Hate SpeechMai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi et al.EMNLP 2021 · 159 citations
  4. Probing LLMs for hate speech detection: strengths and vulnerabilitiesSarthak Roy, Ashish Harshavardhan, Animesh Mukherjee, Punyajoy SahaEMNLP Findings 2023 · 23 citations
  5. Unveiling the Implicit Toxicity in Large Language ModelsJiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang et al.EMNLP 2023 · 21 citations

Related papers

  1. Can Large Language Models Capture Dissenting Human Voices?Noah Lee, Na An, James ThorneEMNLP 2023 · 8 citations
  2. HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech Detection via Large Language ModelsHuy Nghiem, Hal Daumé IIIEMNLP Findings 2024 · 10 citations
  3. Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of OffensivenessShayan Alipour, Indira Sen, Mattia Samory, Tanushree MitraACL Findings 2025 · 2 citations
  4. Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' DisagreementElisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini et al.EMNLP 2021 · 2 citations
  5. Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech DetectionMin Zhang, Jianfeng He, Taoran Ji, Chang-Tien LuACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines