FairPrism: Evaluating Fairness-Related Harms in Text Generation

It is critical to measure and mitigate fairnessrelated harms caused by AI text generation systems, including stereotyping and demeaning harms. To that end, we introduce FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. FairPrism's annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications. The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the "speaker" is an AI system. Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairnessrelated harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods, both of which we illustrate through case studies. Finally, the process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems. Real Toxicity Prompts BOLD ToxiGen Social Bias Frames Our work: FairPrism Text source AI AI AI Human AI Label source (human or classifier) Classifier Classifier Classifier (792 human) Human Human Separates subtypes of harm within toxicity/hate speech? (3.1) No No No No Yes Contextualizes AI responses? (3) No Yes No N/A Yes Identifies target group harmed? (3.2) No No Yes Yes Yes Includes disaggregated data? (3.3) No No No Yes Yes Examines AI-specific harms? (3.4) No No No No Yes