Lune

EMNLP Findings2024

Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning

Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md. Tahmid Rahman Laskar, Mir Tafseer Nayeem, Enamul Hoque

Abstract

Natural language is a powerful complementary modality of communication for data visualizations, such as bar and line charts.To facilitate chart-based reasoning using natural language, various downstream tasks have been introduced recently such as chart question answering, chart summarization, and fact-checking with charts.These tasks pose a unique challenge, demanding both vision-language reasoning and a nuanced understanding of chart data tables, visual encodings, and natural language instructions.Despite the recent success of Large Language Models (LLMs) across diverse NLP tasks, their abilities and limitations in the realm of data visualization remain under-explored, possibly due to their lack of multi-modal capabilities.To bridge the gap, this paper presents one of the first comprehensive evaluations of the recently developed large vision language models (LVLMs) for chart understanding and reasoning tasks.Our evaluation includes a comprehensive assessment of both closed and opensourced LVLMs across five major chart reasoning tasks.Furthermore, we perform a qualitative evaluation of LVLMs' performance on a diverse range of charts, aiming to provide a thorough analysis.Our findings reveal that while LVLMs demonstrate impressive abilities in generating fluent texts covering high-level data insights, they also encounter common problems like hallucinations, factual errors, and data bias.We highlight the key strengths and limitations of LVLMs in chart comprehension tasks, offering insights for future research 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework cites Visual Instruction Tuning
  • PAL: Program-aided Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Do Vision-Language Models Really Understand Visual Language? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Probing the Visualization Literacy of Vision Language Models: The Good, the Bad, and the Ugly cites Visual Instruction Tuning
  • SceneLoom: Communicating Data with Scene Context cites Visual Instruction Tuning
  • SceneLoom: Communicating Data with Scene Context cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMsShmuel Berman, Jia DengNeurIPS 2025 · 7 citations
  2. Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training FrameworkYuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng et al.CHI 2026 · 1 citation
  3. Probing the Visualization Literacy of Vision Language Models: The Good, the Bad, and the UglyLianghan Dong, Anamaria CrisanIEEE VIS 2025 · 1 citation
  4. SceneLoom: Communicating Data with Scene ContextLin Gao, Leixian Shen, Yuheng Zhao, Jiexiang Lan et al.IEEE VIS 2025 · 1 citation
  5. Do Vision-Language Models Really Understand Visual Language?Yifan Hou, Buse Giledereli, Yilei Tu, Mrinmaya SachanICML 2025

Builds on25

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  5. PAL: Program-aided Language ModelsLuyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon et al.ICML 2023 · 700 citations

Related papers

  1. FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language ModelsDong Shu, Haoyang Yuan, Yuchen Wang, Yanguang Liu et al.ACL 2026 · 11 citations
  2. ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question AnsweringAhmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj et al.ACL Findings 2025 · 21 citations
  3. Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from TextMizanur Rahman, Md. Tahmid Rahman Laskar, Shafiq Joty, Enamul HoqueEMNLP 2025 · 1 citation
  4. ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question AnsweringJingxuan Wei, Nan Xu, Junnan Zhu, Yanni Hao et al.EMNLP 2025 · 6 citations
  5. An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy TasksAlexander Bendeck, John T. StaskoIEEE VIS 2024 · 40 citations
Dusk over the sea between two cliffs drawn in fine vertical lines