Is Having Rationales Enough? Rethinking Knowledge Enhancement for Multimodal Hateful Meme Detection

Hateful memes are prevalent on the Internet, raising the urgent need for effective detection.Given their implicit nature, incorporating rationales with background knowledge is crucial for enhancing model understanding.However, existing methods often suffer from limited quality of external rationales and misalignment with original meme information.These challenges hinder model comprehension, leading to reduced accuracy and explainability.To address these challenges, we propose a Multimodal Multi-agent Knowledge Enhanced (M2KE) framework for hateful meme detection.M2KE introduces a multi-agent rationale discovery mechanism to extract high-quality rationales relevant to meme content and an adaptive knowledge interaction mechanism to ensure alignment between original meme information and external rationales.Specifically, multi-agent rationale discovery mechanism improves the reliability of rationales by collaboratively verifying and refining them with multiple agents, supported by large language models (LLMs) due to their extensive knowledge.And adaptive knowledge interaction mechanism uses information entropy to dynamically balance the model's attention between original meme information and external rationales, preventing over-reliance on rationales and enabling a more comprehensive understanding.Experimental results on three datasets demonstrate that M2KE significantly outperforms existing models.Further analysis underscores the importance of effectively integrating accurate rationales to enhance model performance.Disclaimer: Samples in this paper may be considered offensive.