MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval

Document Understanding is a foundational AI capability with broad applications, and Document Question Answering (DocQA) is a key evaluation task.Traditional methods convert the document into text for processing by Large Language Models (LLMs), but this process strips away critical multi-modal information like figures.While Large Vision-Language Models (LVLMs) address this limitation, their constrained input size makes multi-page document comprehension infeasible.Retrievalaugmented generation (RAG) methods mitigate this by selecting relevant pages, but they rely solely on semantic relevance, ignoring logical connections between pages and the query, which is essential for reasoning.To this end, we propose MoLoRAG, a logicaware retrieval framework for multi-modal, multi-page document understanding.By constructing a page graph that captures contextual relationships between pages, a lightweight VLM performs graph traversal to retrieve relevant pages, including those with logical connections often overlooked.This approach combines semantic and logical relevance to deliver more accurate retrieval.After retrieval, the top-K pages are fed into arbitrary LVLMs for question answering.To enhance flexibility, MoLoRAG offers two variants: a training-free solution for easy deployment and a fine-tuned version to improve logical relevance checking.Experiments on four DocQA datasets demonstrate average improvements of 9.68% in accuracy over LVLM direct inference and 7.44% in retrieval precision over baselines.Codes and datasets are released at https://github.com/WxxShirley/MoLoRAG. Question How many days with overflow do Outfall 002A (Southwest Hoboken) and Outfall 005A (Central Hoboken) have in total?