EMNLP Findings2024
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models
Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan
Abstract
Existing evaluation benchmarks for foundation models in understanding scientific literature predominantly focus on single-document, text-only tasks. Such benchmarks often do not adequately represent the complexity of research workflows, which typically also involve interpreting non-textual data, such as figures and tables, and gathering information across multiple documents and related literature. To address this gap, we introduce M3SCIQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3SCIQA consists of 1,452 expertannotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SCIQA, we conduct a comprehensive evaluation of 18 frontier foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis. "The results presented in the Information Value paper are intriguing. The figure clearly indicates that the DialoGPT family of models exhibits the most negative Spearman's Correlation Coefficient compared to all other models." 1 Reading an anchor paper Locality-Specific Question: "Which model exhibits the most negative Spearman's Correlation Coefficient?" Locality-Specific Reasoning: DialoGPT Large exhibits the most negative Spearman's Correlation Coefficient.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Visual Instruction Tuning
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Cited by top-tier papers10
- SemBench: A Benchmark for Semantic Query Processing EnginesJiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong et al.VLDB 2026 · 31 citations
- SciVer: Evaluating Foundation Models for Multimodal Scientific Claim VerificationChengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan et al.ACL 2025 · 8 citations
- Doc-Researcher: A Unified System for Multimodal Document Parsing and Deep ResearchKuicai Dong, Shurui Huang, Fangda Ye, Wei Han et al.WWW 2026 · 4 citations
- PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&AAnna Martin-Boyle, Cara A. C. Leckey, Martha Brown, Harmanpreet KaurCHI 2026 · 3 citations
- AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level EvaluationTiancheng Huang, Ruisheng Cao, Yuxin Zhang, Zhangyi Kang et al.ICLR 2026 · 1 citation
Builds on20
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
- MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
Related papers
- Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific PapersYilun Zhao, Chengye Wang, Chuhan Li, Arman CohanACL Findings 2025 · 3 citations
- PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific PapersLei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao et al.ACL Findings 2026
- SciDQA: A Deep Reading Comprehension Dataset over Scientific PapersShruti Singh, Nandan Sarkar, Arman CohanEMNLP 2024 · 2 citations
- SciMDR: Advancing Scientific Multimodal Document ReasoningZiyu Chen, Yilun Zhao, Chengye Wang, Rilyn Han et al.ACL 2026 · 1 citation
- M³-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question AnsweringJiatong Ma, Longteng Guo, Yuchen Liu, Zijia Zhao et al.ACL 2026
