Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models

Recent advancements in dialogue systems have highlighted the significance of integrating multimodal responses, which enable conveying ideas through diverse modalities rather than solely relying on text-based interactions. This enrichment not only improves overall communicative efficacy but also enhances the quality of conversational experiences. However, existing methods for dialogue-to-image retrieval face limitations due to the constraints of pretrained vision language models (VLMs) in comprehending complex dialogues accurately. To address this, we present a novel approach leveraging the robust reasoning capabilities of large language models (LLMs) to generate precise dialogue-associated visual descriptors, facilitating seamless connection with images. Extensive experiments conducted on benchmark data validate the effectiveness of our proposed approach in deriving concise and accurate visual descriptors, leading to significant enhancements in dialogue-to-image retrieval performance. Furthermore, our findings demonstrate the method's generalizability across diverse visual cues, various LLMs, and different datasets, underscoring its practicality and potential impact in real-world applications. 1 Dialogue context B: how are you doing? A: I'm doing good. Just out at a restaurant taking pictures for customers. B: congratulations A: It's hilarious watching people try to use chopsticks B: i'm really happy for you friend B: yeah, its really funny A: Yeah, it's better than most gigs I get B: even i still try to try to find a way around that thing A: I give up and ask for a fork. I want that rice in my mouth!!!!! A: (share a photo) Ground-truth Retrieved top-1 Dialogue-associated visual cues • main subject: customers • foreground objects: chopsticks, table, food • background scene: restaurant • events: eating food