ACL Findings2025
Just KIDDIN' : Knowledge Infusion and Distillation for Detection of INdecent Memes
Rahul Garg, Trilok Padhi, Hemang Jain, Ugur Kursuncu, Ponnurangam Kumaraguru
Abstract
Detecting toxicity in online multimodal environments, such as memes, remains a challenging task due to the complex contextual connections across modalities (e.g., text and visual), which demand both common-sense reasoning and contextual awareness. To bridge this gap, we propose a hybrid neurosymbolic framework that unifies (1) distillation of implicit contextual knowledge (e.g., sarcasm, cultural references) from Large Vision-Language Models (LVLMs) and ( 2) infusion of explicit relational semantics through sub-graphs from Knowledge Graphs (KGs). Experimental results on two benchmark datasets show the superior performance of our approach, Knowledge-Infused Distilled Vision-Language Model (KID-VLM), over the state-ofthe-art baselines across AUC and F1, with improvements of 0.5%, and 10.6%, respectively, in HatefulMemes Benchmark across variants. Further, KID-VLM demonstrates better generalizability and achieves the best performance across all baselines in the HarMeme Dataset with a 6.3% and 3.2% in F1 and AUC. Given the contextual complexity of the toxicity detection, KID-VLM showcases the significance of learning compact models ( 500M parameters) from both explicit (i.e., KG) and implicit (i.e., LVLMs) contextual cues incorporated through a hybrid neurosymbolic approach. Our codes and pretrained models are publicly available 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets cites Learning Transferable Visual Models From Natural Language Supervision
- Prompting for Multimodal Hateful Meme Classification cites MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
- Prompting for Multimodal Hateful Meme Classification cites Learning Transferable Visual Models From Natural Language Supervision
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds on12
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their TargetsShraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar et al.EMNLP Findings 2021 · 142 citations
- Multimodal Learning with Incomplete Modalities by Knowledge DistillationQi Wang, Liang Zhan, Paul M. Thompson, Jiayu ZhouKDD 2020 · 80 citations
- Prompting for Multimodal Hateful Meme ClassificationRui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, Jing JiangEMNLP 2022 · 68 citations
Related papers
- Miko: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense DiscoveryFeihong Lu, Weiqi Wang, Yangyifei Luo, Ziqin Zhu et al.ACM MM 2024 · 11 citations
- Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal HateAustin Botelho, Scott Hale, Bertie VidgenACL Findings 2021 · 14 citations
- Invariant Meets Specific: A Scalable Harmful Memes Detection FrameworkChuanpeng Yang, Fuqing Zhu, Jizhong Han, Songlin HuACM MM 2023 · 9 citations
- Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language ModelsHongzhan Lin, Ziyang Luo, Jing Ma, Long ChenEMNLP Findings 2023 · 23 citations
- ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs: ShieldVLMShiyao Cui, Qinglin Zhang, Xuan Ouyang, Renmiao Chen et al.ACM MM 2025 · 3 citations
