ACL Findings2023
COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements
Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D. Hwang, Swabha Swayamdipta, Maarten Sap
Abstract
Warning: This paper contains content that may be offensive or upsetting.
Understanding the harms and offensiveness of statements requires reasoning about the social and situational context in which statements are made. For example, the utterance "your English is very good" may implicitly signal an insult when uttered by a white man to a nonwhite colleague, but uttered by an ESL teacher to their student would be interpreted as a genuine compliment. Such contextual factors have been largely ignored by previous approaches to toxic language detection. We introduce COBRA frames, the first context-aware formalism for explaining the intents, reactions, and harms of offensive or biased statements grounded in their social and situational context. We create COBRACORPUS, a dataset of 33k potentially offensive statements paired with machine-generated contexts and free-text explanations of offensiveness, implied biases, speaker intents, and listener reactions.
To study the contextual dynamics of offensiveness, we train models to generate COBRA explanations, with and without access to the context. We find that explanations by context-agnostic models are significantly worse than by contextaware ones, especially in situations where the context inverts the statement's offensiveness (29% accuracy drop). Our work highlights the importance and feasibility of contextualized NLP by modeling social factors.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers16
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity TheoryNiloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov et al.ICLR 2024 · 198 citations
- From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language ModelsJulia Mendelsohn, Ronan Le Bras, Yejin Choi, Maarten SapACL 2023 · 14 citations
- "Fifty Shades of Bias": Normative Ratings of Gender Bias in GPT Generated English TextRishav Hada, Agrima Seth, Harshita Diddee, Kalika BaliEMNLP 2023 · 10 citations
- A Comprehensive Framework to Operationalize Social Stereotypes for Responsible AI EvaluationsAida Mostafazadeh Davani, Sunipa Dev, Héctor Pérez-Urbina, Vinodkumar PrabhakaranEMNLP 2025 · 6 citations
- Words Like Knives: Backstory-Personalized Modeling and Detection of Violent CommunicationJocelyn J. Shen, Akhila Yerukola, Xuhui Zhou, Cynthia Breazeal et al.EMNLP 2025 · 4 citations
Builds on13
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant et al.EMNLP Findings 2020 · 260 citations
- Social Simulacra: Creating Populated Prototypes for Social Computing SystemsJoon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris et al.UIST 2022 · 192 citations
- Latent Hatred: A Benchmark for Understanding Implicit Hate SpeechMai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi et al.EMNLP 2021 · 159 citations
- Human-AI Collaboration via Conditional Delegation: A Case Study of Content ModerationVivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao et al.CHI 2022 · 135 citations
Related papers
- Social Bias Frames: Reasoning about Social and Power Implications of LanguageMaarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky et al.ACL 2020 · 16 citations
- A Dataset for Investigating the Impact of Context for Offensive Language Detection in TweetsMusa Ihtiyar, Ömer Özdemir, Mustafa Erengül, Arzucan ÖzgürEMNLP Findings 2023 · 2 citations
- Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social RelationshipsDavid Jurgens, Agrima Seth, Jackson Sargent, Athena Aghighi et al.ACL 2023 · 4 citations
- KOLD: Korean Offensive Language DatasetYounghoon Jeong, Juhyun Oh, Jongwon Lee, Jaimeen Ahn et al.EMNLP 2022 · 41 citations
- Analyzing Norm Violations in Live-Stream ChatJihyung Moon, Dong-Ho Lee, Hyundong Cho, Woojeong Jin et al.EMNLP 2023 · 5 citations
