EMNLP Findings2024
Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge
Young-Jun Lee, Dokyong Lee, Junyoung Youn, Kyeongjin Oh, Byungsoo Ko, Jonghwan Hyeon, Ho-Jin Choi
Abstract
Humans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However, existing works focus on (1) image-sharing behavior in singular sessions, leading to limited long-term social interaction, and (2) a lack of personalized image-sharing behavior. In this work, we introduce STARK , a large-scale long-term multi-modal conversation dataset that covers a wide range of social personas in a multi-modality format, time intervals, and images. To construct STARK automatically, we propose a novel multi-modal contextualization framework, MCU, that generates longterm multi-modal dialogue distilled from Chat-GPT and our proposed Plan-and-Execute image aligner. Using our STARK, we train a multimodal conversation model, ULTRON 7B, which demonstrates impressive visual imagination ability. Furthermore, we demonstrate the effectiveness of our dataset in human evaluation. We make our source code and dataset publicly available 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions cites Visual Instruction Tuning
- Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models cites Visual Instruction Tuning
- MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers3
- MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with ResistanceSubin Kim, Hoonrae Kim, Jihyun Lee, Yejin Jeon et al.EMNLP 2025 · 6 citations
- MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language ModelsYoung-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang et al.ICCV 2025 · 2 citations
- Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic InteractionsJihyoung Jang, Minwook Bae, Minji Kim, Dilek Hakkani-Tür et al.ACL 2025
Builds on35
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations
Related papers
- LongMP-Bench: A Benchmark for Multimodal Persona Understanding in Long-Term DialoguesZhuoqun Li, Zhaopei Huang, Wenxuan Wang, Qin JinACL Findings 2026
- Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face ConversationSe Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim et al.ACL 2024
- Integrating Stickers into Multimodal Dialogue Summarization: A Novel Dataset and Approach for Enhancing Social Media InteractionYuanchen Shi, Fang KongACM MM 2024 · 7 citations
- TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real WorldHongpeng Lin, Ludan Ruan, Wenke Xia, Peiyu Liu et al.ACM MM 2023 · 10 citations
- Multimodal Multi-turn Conversation Stance Detection: A Challenge Dataset and Effective ModelFuqiang Niu, Zebang Cheng, Xianghua Fu, Xiaojiang Peng et al.ACM MM 2024 · 13 citations
