EMNLP Findings2023
Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models
Abstract
The age of social media is rife with memes. Understanding and detecting harmful memes pose a significant challenge due to their implicit meaning that is not explicitly conveyed through the surface text and image. However, existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner but ignore indepth cognition of the meme text and image. In this paper, we attempt to detect harmful memes based on advanced reasoning over the interplay of multimodal information in memes. Inspired by the success of Large Language Models (LLMs) on complex reasoning, we first conduct abductive reasoning with LLMs. Then we propose a novel generative framework to learn reasonable thoughts from LLMs for better multimodal fusion and lightweight fine-tuning, which consists of two training stages: 1) Distill multimodal reasoning knowledge from LLMs; and 2) Fine-tune the generative framework to infer harmfulness. Extensive experiments conducted on three meme datasets demonstrate that our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models cites Language Models are Few-Shot Learners
- CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models cites Learning Transferable Visual Models From Natural Language Supervision
- CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models cites Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
- CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Segment Anything cites Language Models are Few-Shot Learners
- Segment Anything cites Learning Transferable Visual Models From Natural Language Supervision
- Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models cites Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
- Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection cites Learning Transferable Visual Models From Natural Language Supervision
- MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection cites Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models cites Language Models are Few-Shot Learners
- Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models cites Learning Transferable Visual Models From Natural Language Supervision
- Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models cites Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers13
- Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language ModelsHongzhan Lin, Ziyang Luo, Wei Gao, Jing Ma et al.WWW 2024 · 43 citations
- MIND: A Multi-agent Framework for Zero-shot Harmful Meme DetectionZiyan Liu, Chunxiao Fan, Haoran Lou, Yuexin Wu et al.ACL 2025 · 15 citations
- CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal ModelsZixin Chen, Hongzhan Lin, Ziyang Luo, Mingfei Cheng et al.ACL 2024 · 10 citations
- Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language ModelsShuai Niu, Jing Ma, Hongzhan Lin, Liang Bai et al.ACL 2025 · 5 citations
- MisVisFix: An Interactive Dashboard for Detecting, Explaining, and Correcting Misleading Visualizations using Large Language ModelsAmit Kumar Das, Klaus MuellerIEEE VIS 2025 · 5 citations
Builds on14
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
Related papers
- Towards Low-Resource Harmful Meme Detection with LMM AgentsJianzhao Huang, Hongzhan Lin, Ziyan Liu, Ziyang Luo et al.EMNLP 2024 · 2 citations
- Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal FusionYi Shi, Wenlong Meng, Zhenyuan Guo, Chengkun Wei et al.AAAI 2026
- DR-HM: Distill-then-Reinforce Training with Cognition-Aware Data Synthesis for Harmful Meme DetectionZihan Cheng, Jianxiang Ma, Xiaocui Yang, Peidong Wang et al.ACL Findings 2026
- Is Having Rationales Enough? Rethinking Knowledge Enhancement for Multimodal Hateful Meme DetectionJunyu Lu, Bo Xu, Xiaokun Zhang, Haohao Zhu et al.SIGIR 2025 · 3 citations
- Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme DetectionFengjun Pan, Xiaobao Wu, Tho Quan, Anh Tuan LuuWWW 2026 · 2 citations
