Lune

ACL Findings2025

Just KIDDIN' : Knowledge Infusion and Distillation for Detection of INdecent Memes

Rahul Garg, Trilok Padhi, Hemang Jain, Ugur Kursuncu, Ponnurangam Kumaraguru

Abstract

Detecting toxicity in online multimodal environments, such as memes, remains a challenging task due to the complex contextual connections across modalities (e.g., text and visual), which demand both common-sense reasoning and contextual awareness. To bridge this gap, we propose a hybrid neurosymbolic framework that unifies (1) distillation of implicit contextual knowledge (e.g., sarcasm, cultural references) from Large Vision-Language Models (LVLMs) and ( 2) infusion of explicit relational semantics through sub-graphs from Knowledge Graphs (KGs). Experimental results on two benchmark datasets show the superior performance of our approach, Knowledge-Infused Distilled Vision-Language Model (KID-VLM), over the state-ofthe-art baselines across AUC and F1, with improvements of 0.5%, and 10.6%, respectively, in HatefulMemes Benchmark across variants. Further, KID-VLM demonstrates better generalizability and achieves the best performance across all baselines in the HarMeme Dataset with a 6.3% and 3.2% in F1 and AUC. Given the contextual complexity of the toxicity detection, KID-VLM showcases the significance of learning compact models ( 500M parameters) from both explicit (i.e., KG) and implicit (i.e., LVLMs) contextual cues incorporated through a hybrid neurosymbolic approach. Our codes and pretrained models are publicly available 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets cites Learning Transferable Visual Models From Natural Language Supervision
  • Prompting for Multimodal Hateful Meme Classification cites MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
  • Prompting for Multimodal Hateful Meme Classification cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  3. MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their TargetsShraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar et al.EMNLP Findings 2021 · 142 citations
  4. Multimodal Learning with Incomplete Modalities by Knowledge DistillationQi Wang, Liang Zhan, Paul M. Thompson, Jiayu ZhouKDD 2020 · 80 citations
  5. Prompting for Multimodal Hateful Meme ClassificationRui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, Jing JiangEMNLP 2022 · 68 citations

Related papers

  1. Miko: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense DiscoveryFeihong Lu, Weiqi Wang, Yangyifei Luo, Ziqin Zhu et al.ACM MM 2024 · 11 citations
  2. Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal HateAustin Botelho, Scott Hale, Bertie VidgenACL Findings 2021 · 14 citations
  3. Invariant Meets Specific: A Scalable Harmful Memes Detection FrameworkChuanpeng Yang, Fuqing Zhu, Jizhong Han, Songlin HuACM MM 2023 · 9 citations
  4. Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language ModelsHongzhan Lin, Ziyang Luo, Jing Ma, Long ChenEMNLP Findings 2023 · 23 citations
  5. ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs: ShieldVLMShiyao Cui, Qinglin Zhang, Xuan Ouyang, Renmiao Chen et al.ACM MM 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines