ACL Findings2025
WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts
Negar Foroutan, Angelika Romanou, Matin Ansaripour, Julian Martin Eisenschlos, Karl Aberer, Rémi Lebret
Abstract
Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). While vision-language large models (VLLMs) have demonstrated improvements across various tasks, their effectiveness in processing long-context vision inputs remains unclear. This paper introduces WikiMixQA, a benchmark comprising 1,000 multiple-choice questions (MCQs) designed to evaluate cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages spanning seven distinct topics. Unlike existing benchmarks, WikiMixQA emphasizes complex reasoning by requiring models to synthesize information from multiple modalities. We evaluate 12 state-of-the-art vision-language models, revealing that while proprietary models achieve ∼70% accuracy when provided with direct context, their performance deteriorates significantly when retrieval from long documents is required. Among these, GPT-4-o is the only model exceeding 50% accuracy in this setting, whereas open-source models perform considerably worse, with a maximum accuracy of 27%. These findings underscore the challenges of long-context, multi-modal reasoning and establish WikiMixQA as a crucial benchmark for advancing document understanding research. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- UNICBench: UNIfied Counting Benchmark for MLLM cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Cited by top-tier papers5
- UNICBench: UNIfied Counting Benchmark for MLLMChenggang Rong, Tao Han, Zhiyuan Zhao, Yaowu Fan et al.CVPR 2026 · 3 citations
- Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and EvaluationWei Zhou, Bolei Ma, Annemarie Friedrich, Mohsen MesgarACL 2026 · 3 citations
- CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop ReasoningJunyoung Sung, Seungwoo Lyu, Minjun Kim, Sumin An et al.CVPR 2026 · 2 citations
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and ChartsXanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin et al.AAAI 2026
- DORA: A Dual-Objective Reinforcement Learning Framework for Effective and Efficient Multimodal Agentic SearchGuangming Qin, Yuhao Deng, Yukun Zhao, Zhenyang Li et al.ACL 2026
Builds on7
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationJianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo et al.ACL Findings 2024 · 762 citations
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningAhmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty et al.ACL Findings 2022 · 389 citations
- Document Understanding Dataset and Evaluation (DUDE)Jordy Van Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewicz et al.ICCV 2023 · 130 citations
- Open Question Answering over Tables and TextWenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang et al.ICLR 2021 · 76 citations
- OpenCQA: Open-ended Question Answering with ChartsShankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan et al.EMNLP 2022 · 26 citations
Related papers
- MMCR: Benchmarking Cross-Source Reasoning in Scientific PapersYang Tian, Zheng Lu, Mingqi Gao, Zheng Liu et al.ICCV 2025
- LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and LocatingChao Deng, Jiale Yuan, Pi Bu, Peijie Wang et al.ACL 2025
- MMTableBench: A Multi-level Multimodal Benchmark for Reasoning and Layout Complexity in Table QAXianjie Wu, Xiaohang Xu, Tingyu Jiang, Jian Yang et al.WWW 2026 · 3 citations
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document UnderstandingKetong Chen, Yuhao Chen, Yang XueAAAI 2026 · 1 citation
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language ModelsFanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu et al.ICLR 2025
