Lune

EMNLP Findings2024

Hope 'The Paragraph Guy' explains the rest : Introducing MeSum, the Meme Summarizer

Anas Anwarul Haq Khan, Tanik Saikh, Arpan Phukan, Asif Ekbal

Abstract

Over the years, memes have evolved into multifaceted narratives on platforms like Instagram, TikTok, and Reddit, blending text, images and audio to amplify humor and engagement. The objective of the task described in this article is to bridge the gap for individuals who may struggle to understand memes due to cultural, geographical, ancillary insights, or relevant exposure constraints, aiming to enhance meme comprehension across diverse audiences. The lack of large datasets for supervised learning and alternatives to resource-intensive vision language models have historically hindered the development of such technology. In this work, we have made strides to overcome these challenges. We introduce "MMD" a Multimodal Meme Dataset comprising 13,494 instances, including 3,134 with audio, rendering it the largest of its kind, with 2.1 times as many samples and 9.5 times as many words in the human annotated meme summary compared to the largest available meme captioning dataset, MemeCap. Our framework, MeSum (Meme Summariser), employs a fusion of Vision Transformer and Large Language Model technologies, providing an efficient alternative to resource-intensive Vision Language Models pioneering the integration of all three modalities, we attain a ROUGE-L score of 0.439, outperforming existing approaches such as zero-shot Gemini, GPT4 Vision, LLaVA and QwenVL which yield scores of 0.259, 0.213, 0.177 and 0.198. We have made our codes and datasets publicly available. 1 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  4. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  5. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal MemesDouwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami et al.NeurIPS 2020 · 1,022 citations

Related papers

  1. MemeCap: A Dataset for Captioning and Interpreting MemesEunjeong Hwang, Vered ShwartzEMNLP 2023 · 14 citations
  2. MetaGPT: A Large Vision-Language Model for Meme Metaphor UnderstandingBo Xu, Chenyuan Wang, Xinyu Chen, Hongfei Lin et al.AAAI 2026
  3. MemeQA: Holistic Evaluation for Meme UnderstandingKhoi P. N. Nguyen, Terrence Li, Derek Lou Zhou, Gabriel Xiong et al.ACL 2025 · 3 citations
  4. Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal FusionYi Shi, Wenlong Meng, Zhenyuan Guo, Chengkun Wei et al.AAAI 2026
  5. MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched ContextualizationShivam Sharma, Ramaneswaran S., Udit Arora, Md. Shad Akhtar et al.ACL 2023 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines