Lune

ACL2025Top-tier venue

Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions

Jihyoung Jang, Minwook Bae, Minji Kim, Dilek Hakkani-Tür, Hyounghun Kim

Abstract

As chatbots continue to evolve toward humanlike, real-world, interactions, multimodality remains an active area of research and exploration. So far, efforts to integrate multimodality into chatbots have primarily focused on image-centric tasks, such as visual dialogue and image-based instructions, placing emphasis on the "eyes" of human perception while neglecting the "ears", namely auditory aspects. Moreover, these studies often center around static interactions that focus on discussing the modality rather than naturally incorporating it into the conversation, which limits the richness of simultaneous, dynamic engagement. Furthermore, while multimodality has been explored in multi-party and multi-session conversations, task-specific constraints have hindered its seamless integration into dynamic, natural conversations. To address these challenges, this study aims to equip chatbots with "eyes and ears" capable of more immersive interactions with humans. As part of this effort, we introduce a new multimodal conversation dataset, Multimodal Multi-Session Multi-Party Conversation (M 3 C), and propose a novel multimodal conversation model featuring multimodal memory retrieval. Our model, trained on the M 3 C, demonstrates the ability to seamlessly engage in long-term conversations with multiple speakers in complex, real-world-like settings, effectively processing visual and auditory inputs to understand and respond appropriately. Human evaluations highlight the model's strong performance in maintaining coherent and dynamic interactions, demonstrating its potential for advanced multimodal conversational agents. 1 1 Our code, dataset, and model are publicly available at https://m3c-dataset.github.io/ .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Generating Images with Multimodal Language Models cites Visual Instruction Tuning
  • Generating Images with Multimodal Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • SALMONN: Towards Generic Hearing Abilities for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on35

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen et al.ICLR 2024 · 557 citations
  5. Generating Images with Multimodal Language ModelsJing Yu Koh, Daniel Fried, Russ SalakhutdinovNeurIPS 2023 · 403 citations

Related papers

  1. Mixed-Session Conversation with Egocentric MemoryJihyoung Jang, Taeyoung Kim, Hyounghun KimEMNLP Findings 2024
  2. MPCHAT: Towards Multimodal Persona-Grounded ConversationJaewoo Ahn, Yeda Song, Sangdoo Yun, Gunhee KimACL 2023 · 4 citations
  3. A New Formula for Sticker Retrieval: Reply with Stickers in Multi-Modal and Multi-Session ConversationBingbing Wang, Yiming Du, Bin Liang, Zhixin Bai et al.AAAI 2025 · 5 citations
  4. Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM AgentsYuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao et al.ACL 2026 · 23 citations
  5. Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense KnowledgeYoung-Jun Lee, Dokyong Lee, Junyoung Youn, Kyeongjin Oh et al.EMNLP Findings 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines