VLIS: Unimodal Language Models Guide Multimodal Language Generation

Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field. However, existing vision-language models face challenges in tasks that require complex linguistic understanding. To address this issue, we introduce Visual-Language models as Importance Sampling weights ( VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves visionlanguage models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation. Named Entities Who is this? Does he care for his family? person Diego Maradona Not sure the man in a suit does Yes soccer player Yes, Michael Corleone does. VLMs Distractors Can ostriches fly? Do chimpanzees have tails? VLMs VLIS GPT Yes No No VLMs GPT Yes No No VLIS VLIS