PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs

Understanding scientific papers requires more than answering isolated questions or summarizing content. It involves an integrated reasoning process that grounds textual and visual information, interprets experimental evidence, synthesizes information across sources, and critically evaluates scientific claims. However, existing benchmarks typically assess these abilities in isolation, making it difficult to evaluate scientific paper understanding as a unified set of interacting cognitive abilities. In this work, we introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers. Paper-Mind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics. It comprises four complementary task families that collectively operationalize distinct cognitive facets of scientific paper reasoning, including multimodal grounding, experimental interpretation, cross-source evidence reasoning, and critical assessment. By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors that are difficult to assess through isolated task evaluations. Extensive experiments on both opensource and closed-source multimodal LLMs reveal consistent performance gaps across tasks, highlighting persistent challenges in integrated scientific reasoning and critique. Our benchmark and dataset are available at https:// github.com/Yanjun-Zhao/PaperMind . * Equal contribution. † Corresponding author. derstanding tasks, including document question answering, summarization, and citation-grounded retrieval (e.g., QASPER (Dasigi et al., 2021); Sci-Fact (Wadden et al., 2020); PaperQA (Lála et al., 2023) ). In parallel, tool-augmented and agentic LLM systems have demonstrated the ability to perform multi-step reasoning by interleaving language generation with external actions such as search, code execution, and document retrieval (e.g., Re-Act (Yao et al., 2023); Toolformer (Schick et al., 2023); AutoGen (Wu et al., 2024) ). These advances suggest the potential for multimodal LLM-based agents to assist with scientific research tasks that require structured reasoning over papers, figures, experiments, and citations. However, existing evaluations of scientific understanding tend to emphasize individual capabilities in relatively isolated and static settings. Current benchmarks (