Seeing Beyond Hallucinations: LLM-based Compositional Information Extraction for Multimodal Reasoning

Advancements in Multimodal Large Language Models (MLLMs) have significantly improved information extraction and retrieval performance. Despite these achievements, MLLMs still suffer from the visual object hallucination problem, where models produce plausible, yet incorrect, or irrelevant content not present in the input data. This issue arises from an over-reliance on ''bag-of-objects'' representations and language priors, leading to inadequate extraction of visual objects, along with their attributes and relationships. Existing methods to mitigate these hallucinations are limited by the significant human labor required and the coarse-grained nature. To overcome these challenges, we introduce Multimodal Contrastive Decoding (MMCD), a novel decoding approach that integrates graph-structured reasoning paths with contrastive decoding. MMCD mitigates object hallucinations induced by language priors and enhances the ability of MLLMs to extract and understand compositional information, without additional training or the usage of external tools. This is achieved by masking key objects in images, constructing perturbed scene graphs of attributes and relationships, then contrasting these with the original image and scene graph. Extensive evaluation across three distinct multimodal compositional reasoning tasks: spatial relationship reasoning, alignment of synthetic image and caption, and fine-grained object attribute understanding, show that MMCD consistently surpasses existing decoding methods when applied to various MLLMs. Moreover, MMCD achieves state-of-the-art performance on multiple benchmarks, including the What's Up, SeeTrue and SugarCrepe datasets.