Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning

The widespread presence of hate speech on the internet, including formats such as text-based tweets and vision-language memes, poses a significant challenge to digital platform safety.Recent research has developed detection models tailored to specific modalities; however, there is a notable gap in transferring detection capabilities across different formats.This study conducts extensive experiments using few-shot in-context learning with large language models to explore the transferability of hate speech detection between modalities.Our findings demonstrate that text-based hate speech examples can significantly enhance the classification accuracy of vision-language hate speech.Moreover, text-based demonstrations outperform vision-language demonstrations in fewshot learning settings.These results highlight the effectiveness of cross-modality knowledge transfer and offer valuable insights for improving hate speech detection systems 1 .* These authors contributed equally to this work. 1GitHub: https://github.com/Social-AI-Studio/ Bridging-Modalities is complicated by copyright issues and increasingly stringent regulations on social platforms.Consequently, the limited availability of vision-language data hampers performance in out-of-distribution cases.In contrast, the abundance and diversity of text-based data offer a potential source for crossmodality knowledge transfer (Hee et al., 2024).Research Objectives.This paper investigates whether text-based hate speech detection capabilities can be transferred to multimodal formats.By leveraging the richness of text-based data, we aim to enhance the detection of vision-language hate speech, addressing current research limitations and improving performance in low-resource settings.Contributions.This study makes the following key contributions: (i) We conduct extensive experiments evaluating the transferability of textbased hate speech detection to vision-language formats using few-shot in-context learning with large language models.(ii) We demonstrate that textbased hate speech examples significantly improve the classification accuracy of vision-language hate speech.(iii) We show that text-based demonstrations in few-shot learning contexts outperform vision-language demonstrations, highlighting the potential for cross-modality knowledge transfer.These contributions address critical gaps in existing research and provide a foundation for developing robust hate speech detection systems. Research QuestionsAs all forms of hate speech share one definition, this study investigates the usefulness of using hate speech from one form, such as text-based hate speech, to classify hate speech in another form, such as vision-language hate speech.Working to-