Lune

EMNLP Findings2024

Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details

Yasaman Razeghi, Ishita Dasgupta, Fangyu Liu, Vinay Ramasesh, Sameer Singh

Abstract

Recent advances in multimodal models show remarkable performance in real-world benchmarks for chart and figure understanding like ChartQA that involve interpreting trends, comparing data points, and extracting insights from visuals. In this paper, we investigate the extent to which these models truly comprehend the underlying information in charts by posing direct, elementary questions about simple features such as axes ranges and values to examine their fundamental visual understanding abilities in the context of charts. Our questions are applied to two sets of figures: synthetic and real-world. The empirical evaluation of 5 popular multimodal models on our dataset reveals shortfalls in understanding charts and figures, contrary to what their performance on complex benchmarks might suggest. For instance, Gemini Pro Vision only achieves 57.9% accuracy on our elementary set of questions on real-world plots, while other popular multimodal models showed similar or less performance. This work highlights an important limitation of current multimodal models, and cautions against overly optimistic interpretations of their abilities based on results of canonical evaluations.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • CogVLM: Visual Expert for Pretrained Language Models cites MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
  • CogVLM: Visual Expert for Pretrained Language Models cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering cites CogVLM: Visual Expert for Pretrained Language Models
  • MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
  • MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
  • MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question AnsweringRachneet Kaur, Nishan Srishankar, Zhen Zeng, Sumitra GaneshACL 2026 · 5 citations

Builds on13

  1. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  2. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
  3. MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
  4. CogVLM: Visual Expert for Pretrained Language ModelsWeihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong et al.NeurIPS 2024 · 858 citations
  5. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningAhmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty et al.ACL Findings 2022 · 389 citations

Related papers

  1. ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question AnsweringAhmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj et al.ACL Findings 2025 · 21 citations
  2. DomainCQA: Crafting Knowledge-Intensive QA from Domain-Specific ChartsYujing Lu, Ling Zhong, Jing Yang, Weiming Li et al.AAAI 2026
  3. OpenCQA: Open-ended Question Answering with ChartsShankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan et al.EMNLP 2022 · 26 citations
  4. Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and RobustnessSrija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu et al.EMNLP Findings 2024 · 5 citations
  5. ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question AnsweringJingxuan Wei, Nan Xu, Junnan Zhu, Yanni Hao et al.EMNLP 2025 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines