MR-RAG: Multimodal Relevance-Aware Retrieval-Augmented Generation for Medical Visual Question Answering

Large Vision Language Models (LVLMs) with retrievalaugmented generation (RAG) are emerging as a main paradigm for processing vision-language medical tasks due to their promising achievements. However, existing approaches exhibit two significant limitations in both retrieval and generation stage: First, during the retrieval stage, most methods typically rely on a single similarity signal to estimate document relevance, ignoring the rich information available in multimodal data, which may fail to accurately retrieve matching content. Second, in the generation stage, retrieved documents are integrated directly and uniformly into the input for LVLMs, without taking into account their varying relevance to the question, which may result in the dilution of crucial information and exacerbate the negative impact of irrelevant content. To address these limitations, we propose MR-RAG, a dual-stage RAG enhancement framework by considering multimodal relevance in both retrieval and generation phases. Specifically, a Multimodal Cooperative Retrieval (MCR) module leverages intra-and cross-modal signals for precise document alignment. Subsequently, an Importance-Aware Information Flow Augmentation (IFA) mechanism adjusts attention paths based on fused relevance to refine answer generation. By bridging retrieval and generation via multimodal signals, MR-RAG significantly improves factual accuracy. Experiments on three medical datasets show our method outperforms stateof-the-art baselines, achieving up to a 6.4% accuracy gain.