ACL Findings2025
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models
Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, Xin Eric Wang
Abstract
Existing Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs, leaving open the question of whether they can handle inconsistencies in real-world, layout-rich content. To bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR) benchmark to assess MLLMs' ability to detect and reason about semantic mismatches in artifacts such as webpages, presentation slides, and posters. MMIR comprises 534 challenging samples, each containing synthetically injected errors across five reasoning-heavy categories: Factual Contradiction, Identity Misattribution, Contextual Mismatch, Quantitative Discrepancy, and Temporal/Spatial Incoherence. We evaluate eight state-of-the-art MLLMs, showing that models with dedicated multimodal reasoning capabilities, such as o1, substantially outperform their counterparts while opensource models remain particularly vulnerable to inconsistency errors. Detailed error analyses further show that models excel in detecting pairwise inconsistencies but struggle with inconsistencies confined to single elements in complex layouts. Probing experiments reveal that single-modality prompting, including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal gains, revealing a key bottleneck in cross-modal reasoning. Our findings highlight the need for advanced multimodal reasoning and point to future research on multimodal inconsistency.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Learning Transferable Visual Models From Natural Language Supervision
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Flamingo: a Visual Language Model for Few-Shot Learning
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Mechanisms of Prompt-Induced Hallucination in Vision-Language Models cites MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models cites Large Language Models are Zero-Shot Reasoners
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
- CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution cites Flamingo: a Visual Language Model for Few-Shot Learning
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding cites Flamingo: a Visual Language Model for Few-Shot Learning
Cited by top-tier papers8
- Mechanisms of Prompt-Induced Hallucination in Vision-Language ModelsWilliam Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov et al.ACL 2026 · 4 citations
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image UnderstandingJuntao Jiang, Jiangning Zhang, Yali Bi, Jinsheng Bai et al.ICLR 2026 · 3 citations
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal InconsistenciesLukas Selch, Yufang Hou, Muhammad Jehanzeb Mirza, Sivan Doveh et al.ICLR 2026 · 2 citations
- CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict ResolutionBaoliang Tian, Yuxuan Si, Jilong Wang, Lingyao Li et al.AAAI 2026 · 2 citations
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language ModelsYijie Tong, Yifan Hou, Shaobo Cui, Antoine Bosselut et al.ICML 2026 · 1 citation
Builds on18
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
Related papers
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language ModelsXingrui Wang, Jiang Liu, Chao Huang, Xiaodong Yu et al.ICLR 2026 · 4 citations
- Unveiling the Tapestry of Consistency in Large Vision-Language ModelsYuan Zhang, Fei Xiao, Tao Huang, Chun-Kai Fan et al.NeurIPS 2024 · 27 citations
- Same Content, Different Answers: Cross-Modal Inconsistency in MLLMsAngela van Sprang, Laurens Samson, Ana Lucic, Erman Acar et al.CVPR 2026 · 8 citations
- MMTabReal: Real-World Benchmark for Multimodal Table UnderstandingPrasham Yatinkumar Titiya, Jainil Trivedi, Chitta Baral, Vivek GuptaACL Findings 2026
- Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language ModelsBoyu Jia, Junzhe Zhang, Huixuan Zhang, Xiaojun WanEMNLP Findings 2025
