V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

What color is the liquid in the glass?

"Sorry, the current visual information is not enough. I need to find the glass first."

"The glass is most likely to appear on the dining table."

"The liquid in the glass is green."

GPT4-V answer: "The liquid in the glass appears to be a shade of pink."

Figure 1. The visual search mechanism enables humans to identify a target within a multitude of stimuli, streamlining the organization of information critical for problem-solving and reasoning. In this work, we explore this core mechanism in the context of MLLMs, addressing its absence, which currently impedes precise visual grounding, especially for high-resolution images. In this example, the VQA LLM could not immediately answer the question, thus activating V * , an LLM-guided visual search process that uses common sense and contextual cues to search for the required details. Throughout this informed search, it builds a visual working memory (VWM), tokenizing the overall context and the areas of interest related to the targets, which are then re-fed to the VQA LLM, enabling it to accurately answer the question.