Lune

ACL Findings2024

SoMeLVLM: A Large Vision Language Model for Social Media Processing

Xinnong Zhang, Haoyu Kuang, Xinyi Mou, Hanjia Lyu, Kun Wu, Siming Chen, Jiebo Luo, Xuanjing Huang, Zhongyu Wei

Abstract

The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities. More information can be viewed at SoMeLVLM.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance cites Visual Instruction Tuning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in InsuranceChenwei Lin, Hanjia Lyu, Xian Xu, Jiebo LuoICCV 2025 · 3 citations
  2. Omni-Attack: Adversarial Attacks on Open-Ended VQA in Black-Box Multimodal LLMsKai Hu, Weichen Yu, Li Zhang, Alexander Robey et al.CVPR 2026

Builds on21

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  5. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations

Related papers

  1. Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
  2. SoMe: A Realistic Benchmark for LLM-based Social Media AgentsDizhan Xue, Jing Cui, Shengsheng Qian, Chuanrui Hu et al.AAAI 2026 · 1 citation
  3. MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
  4. VIBE: Can a VLM Read the Room?Tania Chakraborty, Eylon Caplan, Dan GoldwasserEMNLP Findings 2025 · 2 citations
  5. MM-SOC: Benchmarking Multimodal Large Language Models in Social Media PlatformsYiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang et al.ACL Findings 2024 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines