ACL Findings2025
MANBench: Is Your Multimodal Model Smarter than Human?
Han Zhou, Qitong Xu, Yiheng Dong, Xin Yang
Abstract
The rapid advancement of Multimodal Large Language Models (MLLMs) has ignited discussions regarding their potential to surpass human performance in multimodal tasks. In response, we introduce MANBench (Multimodal Ability Norms Benchmark), a bilingual benchmark (English and Chinese) comprising 1,314 questions across nine tasks, spanning knowledge-based and non-knowledge-based domains. MANBench emphasizes intuitive reasoning, seamless cross-modal integration, and real-world complexity, providing a rigorous evaluation framework.
Through extensive human experiments involving diverse participants, we compared human performance against state-of-the-art MLLMs.
The results indicate that while MLLMs excel in tasks like Knowledge and Text-Image Understanding, they struggle with deeper crossmodal reasoning tasks such as Transmorphic Understanding, Image Consistency, and Multiimage Understanding. Moreover, both humans and MLLMs face challenges in highly complex tasks like Puzzles and Spatial Imagination.
MANBench highlights the strengths and limitations of MLLMs, revealing that even advanced models fall short of achieving human-level performance across many domains. We hope MANBench will inspire efforts to bridge the gap between MLLMs and human multimodal capabilities. The code and dataset are available at https://github.com/micdz/MANBench.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Improved Baselines with Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Improved Baselines with Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Are We on the Right Way for Evaluating Large Vision-Language Models? cites Improved Baselines with Visual Instruction Tuning
- Are We on the Right Way for Evaluating Large Vision-Language Models? cites Learning Transferable Visual Models From Natural Language Supervision
- Are We on the Right Way for Evaluating Large Vision-Language Models? cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Builds on7
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- Are We on the Right Way for Evaluating Large Vision-Language Models?Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang et al.NeurIPS 2024 · 1,029 citations
- Improved Baselines with Visual Instruction TuningHaotian Liu, Chunyuan Li, Yuheng Li, Yong Jae LeeCVPR 2024
Related papers
- OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop ReasoningSeunghee Kim, Ingyu Bang, Seokgyu Jang, Changhyeon Kim et al.ACL Findings 2026
- MRBench: A Multi-Image Reasoning Benchmark with Adaptive Knowledge RetrievalWenxi Huang, Xiaojun Chen, Qin Zhang, Ting Wan et al.ACM MM 2025
- GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?Mohammad Reza Taesiri, Tianjun Feng, Cor-Paul Bezemer, Anh NguyenCVPR 2024 · 7 citations
- MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific ProblemsXinwu Ye, Chengfan Li, Siming Chen, Wei Wei et al.ACL Findings 2025 · 1 citation
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied AgentsRui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao et al.ICML 2025
