ACL Findings2025
Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
Yanbei Jiang, Yihao Ding, Chao Lei, Jiayang Ao, Jey Han Lau, Krista A. Ehinger
Abstract
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception.Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process.Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail.To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark based on RAVEN, to assess reasoning across varying levels of complexity.Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps.Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes.We conduct comprehensive experiments on MultiStAR using 17 representative close-source and opensource MLLMs.The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.The dataset and code are available at https://github.com/YanbeiJiang/MultiStARSingle-Step AVR Q: Which panel (1-8) should be placed in the empty box to complete the pa�ern?Logical Chain Ans: "1" Problem Matrix Answer Set Q: Iden�fy the rule about the number of objects.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites Visual Instruction Tuning
- What matters when building vision-language models? cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- What matters when building vision-language models? cites Visual Instruction Tuning
- What matters when building vision-language models? cites MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Cited by top-tier papers1
Builds on9
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- TransVG: End-to-End Visual Grounding with TransformersJiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou et al.ICCV 2021 · 468 citations
- What matters when building vision-language models?Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor SanhNeurIPS 2024 · 401 citations
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIXiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang et al.CVPR 2024 · 213 citations
Related papers
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual ReasoningHao Yan, Xingchen Liu, Hao Wang, Zhenbiao Cao et al.ICLR 2026 · 7 citations
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language ModelsWeiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen et al.ICLR 2026 · 103 citations
- LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMsOmkar Thawakar, Dinura Dissanayake, Ketan Pravin More, Ritesh Thawkar et al.ACL Findings 2025 · 19 citations
- Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric ReasoningLongteng Guo, Yifan Wang, Pengkang Huo, Tailai Chen et al.ACL Findings 2026
- M³CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-ThoughtQiguang Chen, Libo Qin, Jin Zhang, Zhi Chen et al.ACL 2024 · 13 citations
