Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science

The emergence of Multimodal Large Language Models (MLLMs) that integrate vision and language modalities has unlocked new potentials for scientific reasoning, outperforming prior benchmarks in both natural language and coding domains. Current materials science evaluation datasets such as MaScQA and SciQA remain largely text-based and fail to capture the visual and research-level analytic complexity required in materials discovery and design. We introduce MatVQA, a scalable benchmark specifically designed to address this gap. Generated via an automated pipeline, MArxivAgent, from recent materials literature, MatVQA features 1325 questions across four critical structure-property-performance (SPP) reasoning tasks. Uniquely, MatVQA employs an iterative process to eliminate textual shortcuts, compelling MLLMs to perform fine-grained, low-level visual analysis of material imagery (e.g., microscopy, diffraction patterns) integrated with multi-step scientific reasoning. Benchmarking 17 open-and closed-source MLLMs on MatVQA reveals substantial gaps in current multimodal reasoning capabilities. MatVQA benchmark data, along with evaluation code, is publicly available in https://anonymous.4open.science/r/matvqa-1E01/README .md to catalyze further research in applying MLLMs to complex materials science problems. Preprint. Under review. MacBench [5] Research Chem&Mat Lab 628 LabBench [25] Research Biology WebQA 181 MicroVQA [13] Research Microscopy Expert 1,042 MatVQA (Ours) Research Materials Paper 1,325 (b) MatVQA benchmark attributes MatVQA feature Value Total questions 1,325 Causal SPP questions 950 Comp SPP questions 112 Hypo SPP questions 256 Quan SPP questions 7 Unique images 378 Unique papers 44 Research areas 32 grain size) and excludes the low-level visual cues-diffraction peaks, defect textures, subtle contrast variations-crucial for authentic reasoning. Additional leakage stems from distractors generated with caption vocabulary, reliance on caption-embedded numerical values, and minimal emphasis on spatial or pixel-level patterns. Eliminating both shortcut classes is therefore imperative: it compels models to ground their answers in fine-grained visual evidence and provides a benchmark that more faithfully measures genuine vision-language competence. To excise these artifacts, MArxivAgent executes an iterative shortcut-elimination loop. After initial question synthesis, an evaluator agent answers the MCQ using (a) only the stem and options and (b) the stem, options plus caption but without the image. Success in either mode triggers a rewriter that removes or rephrases the incriminating text while a consistency checker enforces fidelity to the original scientific claim. By progressively eliminating both language and caption shortcuts, we elevate the benchmark from coarse-to fine-grained difficulty: solving the final questions requires precise, low-level visual scrutiny (e.g., counting diffraction spots, discerning lattice fringes) coupled with multi-hop scientific reasoning. This refinement is essential for measuring the true multimodal competence that front-line materials research demands. We evaluated a suite of 17 open-and closed-source MLLMs on MatVQA and compared the performance of a select subset against human experts and vision-language model baselines. In summary, our key contributions are: • We release MatVQA, the first benchmark designed to evaluate research-level multimodal reasoning in varies domain of materials science. • We propose four Structure-Property-Performance (SPP) tasks that encapsulate core scientific inquiries regarding material structure, properties, and performance. • We design MArxivAgent, a fully automated, three-stage pipeline that (i) extracts reasoning paths from scientific literature, (ii) iteratively eliminates language shortcuts, and (iii) subsequently removes caption shortcuts, producing high-difficulty, visually grounded MCQs. This work aims to foster the development of MLLMs capable of contributing meaningfully to materials science research by providing a challenging and relevant evaluation standard. 2 Related Work MLLM Reasoning Benchmarks Recent work has introduced a range of benchmarks that probe how well multimodal large language models (MLLMs) integrate visual and textual reasoning. MATHVISTA [29] tests fine-grained visual understanding and compositional math reasoning with thousands of expert-designed problems that expose the gap between today's models and human mathematicians. Similarly, benchmarks focusing on code reasoning, such as CRUXEval [17] evaluates input-output prediction for Python functions, CodeMMLU [33] measures code comprehension across multiple languages and domains, and CRQBench [16] derives reasoning questions from real-world code reviews. There are also