Benchmarking Fine-Grained Error Detection in Multimodal Reasoning

Multimodal Process Reward Models (MPRMs) have emerged as a pivotal framework for enhancing the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, the research community currently lacks a dedicated benchmark to rigorously assess the error discernment capabilities of these models. To address this gap, we introduce PRMBench-V, a novel benchmark specifically designed to evaluate MPRMs' proficiency in detecting erroneous reasoning steps across diverse error categories. Leveraging a semi-automated annotation pipeline augmented with human verification, we construct a comprehensive dataset comprising 907 unique queries, each annotated with nine distinct error types, resulting in 8,163 test cases with fine-grained steplevel error labels. Through extensive experiments involving over 16 open-and closedsource models, we uncover several key findings: (1) even the strongest existing MPRMs achieve only 30% accuracy in error identification; (2) while partial error detection achieves moderate precision and recall ( 60%), overall accuracy remains low ( 20%); and (3) benchmark scores exhibit a strong correlation with downstream task performance gains (r=0.86). Furthermore, we demonstrate that PRMBench-V can inform the development of more robust MPRMs: by introducing the Bayesian Rater Reliability Process Reward Model (BR 2 -PRM), we achieve up to a 4.8% performance improvement through test-time scaling. We believe that PRMBench-V will serve as a valuable resource for advancing MPRM research, enabling more rigorous evaluation and fostering the development of models with finegrained multimodal reasoning capabilities. Based on the previous reasoning, please provide the final answer. You are a helpful AI assistant that is very good at reasoning. Please solve the following problems step by step.