Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets Question: Which animals eats the fruits / leaves of this plant?

Vanilla RAG: The fruits and leaves of Monarda fistulosa//...//many insects and small birds may utilize the nectar and pollen from the flowers for food.

Based on the information provided in the context, the article does not specify which animals. //....// Additionally, herbivorous mammals are said to avoid the foliage due to its odor.

The passage does not provide//...//The known pollinators for this plant are bees, butterflies, and hummingbirds, as well as DOUFIOREA MONARDAE, which itself is a pollinator.

Question: How many meters high can this plant grow?

Vanilla RAG: The image shows Fuchsia magellanica, which typically grows to heights of 1 to 2 meters.

This plant can grow up to 10 meters in height in frost-free climates.

The plant Fuchsia magellanica can grow up to 3.0 meters high in frost-free climates and up to 1.5 meters high in colder conditions.