A Comprehensive Framework to Operationalize Social Stereotypes for Responsible AI Evaluations
Aida Mostafazadeh Davani, Sunipa Dev, Héctor Pérez-Urbina, Vinodkumar Prabhakaran
Abstract
Societal stereotypes are at the center of a myriad of responsible AI interventions targeted at reducing the generation and propagation of potentially harmful outcomes. While these efforts are much needed, they tend to be fragmented and often address different parts of the issue without adopting a unified or holistic approach to social stereotypes and how they impact various parts of the machine learning pipeline. As a result, current interventions fail to capitalize on the underlying mechanisms that are common across different types of stereotypes, and to anchor on particular aspects that are relevant in certain cases. In this paper, we draw on social psychological research and build on NLP data and methods, to propose a unified framework to operationalize stereotypes in generative AI evaluations. Our framework identifies key components of stereotypes that are crucial in AI evaluation, including the target group, associated attribute, relationship characteristics, perceiving group, and context. We also provide considerations and recommendations for its responsible use. CONTENT WARNING: This paper contains examples of stereotypes that may be offensive.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on15
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language TechnologiesSunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian et al.EMNLP 2021 · 113 citations
- Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language ModelsMyra Cheng, Esin Durmus, Dan JurafskyACL 2023 · 89 citations
- Language (Technology) is Power: A Critical Survey of "Bias" in NLPSu Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna M. WallachACL 2020 · 68 citations
- KOLD: Korean Offensive Language DatasetYounghoon Jeong, Juhyun Oh, Jongwon Lee, Jaimeen Ahn et al.EMNLP 2022 · 41 citations
Related papers
- Rethinking Research on Stereotypes: An Analysis through Social Psychological and Computational PerspectivesKaustubh Shivshankar Shejole, Pushpak BhattacharyyaACL Findings 2026
- Analyzing Stereotypes in Generative Text Inference TasksAnna Sotnikova, Yang Trista Cao, Hal Daumé III, Rachel RudingerACL Findings 2021 · 11 citations
- A Scoping Review of Gender Stereotypes in Artificial IntelligenceWen Duan, Lingyuan Li, Guo Freeman, Nathan J. McNeeseCHI 2025 · 15 citations
- Auditing LLM Responses to Harmful Stereotypes Targeting Mental Health GroupsArka Dutta, Rijul Magu, Sean Kim, Seohee Yoon et al.ACL Findings 2026
- Association of Objects May Engender Stereotypes: Mitigating Association-Engendered Stereotypes in Text-to-Image GenerationJunlei Zhou, Jiashi Gao, Xiangyu Zhao, Xin Yao et al.NeurIPS 2024 · 5 citations
