Lune

ICLR2025Top-tier venue

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo

Abstract

We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

50 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Language Models are Few-Shot Learners
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Visual Instruction Tuning
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Flamingo: a Visual Language Model for Few-Shot Learning
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Visual Instruction Tuning cites Language Models are Few-Shot Learners
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
  • Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding cites PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
  • Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding cites Visual Instruction Tuning
  • Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning cites Visual Instruction Tuning
  • LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning cites Language Models are Few-Shot Learners
  • Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers50

  1. MMSI-Bench: A Benchmark for Multi-Image Spatial IntelligenceSihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang et al.ICLR 2026 · 195 citations
  2. LLaDA-V: Large Language Diffusion Models with Visual Instruction TuningZebin You, Shen Nie, Xiaolu Zhang, JUN ZHOU et al.CVPR 2026 · 154 citations
  3. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and GroundingChristopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park et al.CVPR 2026 · 144 citations
  4. PerceptionLM: Open-Access Data and Models for Detailed Visual UnderstandingJang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras et al.NeurIPS 2025 · 97 citations
  5. VisMem: Latent Vision Memory Unlocks Potential of Vision-Language ModelsXinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen et al.CVPR 2026 · 30 citations

Builds on24

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  5. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations

Related papers

  1. MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language ModelsFanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu et al.ICLR 2025
  2. MRBench: A Multi-Image Reasoning Benchmark with Adaptive Knowledge RetrievalWenxi Huang, Xiaojun Chen, Qin Zhang, Ting Wan et al.ACM MM 2025
  3. Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and ImageYushi Hu, Reyhane Askari Hemmat, Melissa Hall, Emily Dinan et al.CVPR 2026 · 18 citations
  4. IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMsDavid Ma, Yuanxing Zhang, Jincheng Ren, Jiawei Guo et al.ICLR 2026 · 5 citations
  5. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIXiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang et al.CVPR 2024 · 213 citations
Dusk over the sea between two cliffs drawn in fine vertical lines