Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering

Knowledge-based visual question answering (VQA) requires external knowledge beyond the image to answer the question. Early studies retrieve required knowledge from explicit knowledge bases (KBs), which often introduces irrelevant information to the question, hence restricting the performance of their models. Recent works have sought to use a large language model (i.e., GPT-3 [3]) as an implicit knowledge engine to acquire the necessary knowledge for answering. Despite the encouraging results achieved by these methods, we argue that they have not fully activated the capacity of GPT-3 as the provided input information is insufficient. In this paper, we present Prophet-a conceptually simple framework designed to prompt GPT-3 with answer heuristics for knowledge-based VQA. Specifically, we first train a vanilla VQA model on a specific knowledgebased VQA dataset without external knowledge. After that, we extract two types of complementary answer heuristics from the model: answer candidates and answer-aware examples. Finally, the two types of answer heuristics are encoded into the prompts to enable GPT-3 to better comprehend the task thus enhancing its capacity. Prophet significantly outperforms all existing state-of-the-art methods on two challenging knowledge-based VQA datasets, OK-VQA and A-OKVQA, delivering 61.1% and 55.7% accuracies on their testing sets, respectively. Q: what fills the balloons? testing sample training samples answer-aware examples Candidates: • candle(0.99) • birthday(0.02) • fire(0.01) ...

helium Please answer the question according to the context and answer candidates . Each answer candidate is associated with a confidence score within a bracket. The true answer may not be included in the candidates. Candidates: • air(0.28) • helium(0.07) • wine(0.03) Candidates: • air(0.69) • helium(0.62) • oxygen(0.04) ... Frozen GPT-3 Model Context: Inflated kites in various shapes float in the air. Question: What chemical makes cats fly? Candidates: air (0.69), helium (0.62), oxygen (0.04) Answer: helium Context: The man is smiling at a birthday cake. Question: What is he about to blow out? Candidates: candle (0.99), birthday (0.02), fire (0.01) Answer: candle Context: a group of children stand around a cake. Question: What fills the balloons? Candidates: air (0.28), helium (0.07), wine (0.03