Maintaining Reasoning Consistency in Compositional Visual Question Answering

A compositional question refers to a question that contains multiple visual concepts (e.g., objects, attributes, and relationships) and requires compositional reasoning to answer. Existing VQA models can answer a compositional question well, but cannot work well in terms of reasoning consistency in answering the compositional question and its sub-questions. For example, a compositional question for an image is: "Are there any elephants to the right of the white bird?" and one of its sub-questions is " Is any bird visible in the scene?". The models may answer "yes" to the compositional question, but "no" to the sub-question. This paper presents a dialog-like reasoning method for maintaining reasoning consistency in answering a compositional question and its sub-questions. Our method integrates the reasoning processes for the sub-questions into the reasoning process for the compositional question like a dialog task, and uses a consistency constraint to penalize inconsistent answer predictions. In order to enable quantitative evaluation of reasoning consistency, we construct a GQA-Sub dataset based on the well-organized GQA dataset. Experimental results on the GQA dataset and the GQA-Sub dataset demonstrate the effectiveness of our method. * corresponding author (a) (b) Q: Is the train to the right or to the left of the black vehicle? (GT: left) Sub-Q1: Is there a vehicle that is black in this image? (GT: yes) Sub-Q2: What color does the vehicle have? (GT: black) left LCGN no green yes Sub-Q3: Do you see a train? (GT: yes) Q: Are there any elephants to the right of the white bird? (GT: yes)