Lune

CVPR Findings2026

MathAll: A Real-World Benchmark for Mathematical Reasoning and Cross-Modal Understanding Evaluation in Omni-MLLMs

Zhilin Lin, Zhihui Zhang, Shiliang Sun, Jing Zhao, Hao Yang

Abstract

Omni-MLLMs have attracted growing attention for their potential to emulate human-like multimodal perception and reasoning. However, existing mathematical reasoning benchmarks are confined to text-only or text-image formats, lacking a comprehensive evaluation of omnimodal capabilities. Moreover, these benchmarks predominantly rely on textbook-style images that differ significantly from the diverse and unstructured visuals encountered in realworld educational and human-computer interaction scenarios. To address these limitations, we propose MathAll, a real-world multimodal mathematics benchmark encompassing text, image, and audio modalities. By incorporating hand-drawn images and explanatory audio, MathAll simulates authentic human reasoning processes and enables comprehensive evaluation of both modality-complementary and modality-integrated reasoning. Extensive experiments demonstrate that current Omni-MLLMs struggle to effectively handle cross-modal inputs, especially when modalities contain imperfect or ambiguous information, and consequently lag behind human-level reasoning. As input complexity increases, models experience difficulties in visual perception and audio understanding, exposing inherent limitations in modality coordination and generalization. Based on these findings, we propose several promising directions for future research and model enhancement.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on21

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  5. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations

Related papers

  1. OmnixR: Evaluating Omni-modality Language Models on Reasoning across ModalitiesLichang Chen, Hexiang Hu, Mingda Zhang, Yiwen Chen et al.ICLR 2025
  2. HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to InteractionZhong Muyan, Erfei Cui, Sen Xing, Weiyun Wang et al.CVPR 2026
  3. VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in VideoHanoona Abdul Rasheed, Abdelrahman M Shaker, Anqi Tang, Muhammad Maaz et al.ICLR 2026 · 16 citations
  4. Probing Audio-Visual Reasoning in Multimodal Language Models through the Lens of AudioKaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang et al.ACL 2026
  5. AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMsYaoting Wang, Ziyi Zhang, Wenming Tu, Shaoxuan Xu et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines