MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems

Recent advances in large language models (LLMs) and vision-language models (LVLMs) have shown promise across many tasks, yet their scientific reasoning capabilities remain untested, particularly in multimodal settings. We present MMSciBench, a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats, with human-annotated difficulty levels, solutions with detailed explanations, and taxonomic mappings. Evaluation of state-of-the-art models reveals significant limitations, with even the best model achieving only 63.77% accuracy and particularly struggling with visual reasoning tasks. Our analysis exposes critical gaps in complex reasoning and visual-textual integration, establishing MMSciBench as a rigorous standard for measuring progress in multimodal scientific understanding. The code for MM-SciBench is open-sourced at GitHub 1 , and the dataset is available at Hugging Face 2 . * Corresponding Authors. Question & Standard Solution Question Question (Single Choice): As shown in the figure, two identical right-angled glass prisms ABC are placed with their AC faces parallel to each other, and between them is a uniform unknown transparent medium. A monochromatic thin light beam O is incident perpendicular to the AB face. ( ) is the possible exit light path in the diagram. Options: A. Any one of the lines 1, 2, 3 (parallel to each other) B. Any one of the lines 4, 5, 6 (parallel to each other) C. Any one of the lines 7, 8, 9 (parallel to each other) D. Only one of the lines 4 or 6 Difficulty Level: 0.7 Domain: Quantum Mechanics Module: Light and Its Applications Chapter: Snell's Law Standard Solution: B Explanation This question primarily tests knowledge of prismrelated problems. Option analysis: According to the problem description, the refractive index of the medium between the two right-angled prisms is unknown. It may be greater than, equal to, or smaller than the refractive index of the glass. The possible light path diagrams are as follows: Therefore, Option B is correct, and Options A, C, and D are incorrect. In conclusion, the correct answer to this question is B. 0 100 Accuracy (%) Applied Mathematical Modeling and Mathematical Inquiry -Mathematical Modeling and Mathematical Inquiry Calculus -Calculus Functions -Functions Functions -