RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RE-ALTOXICITYPROMPTS, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widelyused toxicity classifier. Using REALTOXICI-TYPROMPTS, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data-or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et al., 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

5 https://github.com/conversationai/ perspectiveapi/blob/master/3-concepts/ score-normalization.md 6 To assess PERSPECTIVE API on human-generated text, the first three authors performed manual judgments of toxicity of a sample of 100 documents from OWTC, and found an 88% pairwise agreement (Pearson ⇢=0.83) with TOXICITY scores. To assess the API on machine-generated text, among 100 generations from GPT-2, our judgments had 80% pairwise agreement and Pearson ⇢=0.65 with TOXICITY. For further model information, we refer the reader to the model card for TOX-ICITY: