Silencing Empowerment, Allowing Bigotry: Auditing the Moderation of Hate Speech on Twitch

Warning: This paper contains content that may be offensive or upsetting. To meet the demands of content moderation, online platforms have resorted to automated systems. Newer forms of real-time engagement (e.g., users commenting on live streams) on platforms like Twitch exert additional pressures on the latency expected of such moderation systems. Despite their prevalence, relatively little is known about the effectiveness of these systems. In this paper, we conduct an audit of Twitch's automated moderation tool (AutoMod) to investigate its effectiveness in flagging hateful content. For our audit, we create streaming accounts to act as siloed test beds, and interface with the live chat using Twitch's APIs to send over 107, 000 comments collated from 4 datasets. We measure AutoMod's accuracy in flagging blatantly hateful content containing misogyny, racism, ableism and homophobia. Our experiments reveal that a large fraction of hateful messages, up to 94% on some datasets, bypass moderation. Contextual addition of slurs to these messages results in 100% removal, revealing AutoMod's reliance on slurs as a moderation signal. We also find that contrary to Twitch's community guidelines, AutoMod blocks up to 89.5% of benign examples that use sensitive words in pedagogical or empowering contexts. Overall, our audit points to large gaps in AutoMod's capabilities and underscores the importance for such systems to understand context effectively. 1 * Equal contribution. 1 Our code and data can be found at https://github. com/weiyinc11/HateSpeechModerationTwitch