Lune

EMNLP Findings2025

ProcVQA: Benchmarking the Effects of Structural Properties in Mined Process Visualizations on Vision-Language Model Performance

Kazi Tasnim Zinat, Saad Mohammad Abrar, Shoumik Saha, Sharmila Duppala, Saimadhav Naga Sakhamuri, Zhicheng Liu

Abstract

Vision-Language Models have shown both impressive capabilities and notable failures in data visualization understanding tasks, but we have limited understanding on how specific properties within a visualization type affect model performance. We present ProcVQA, a benchmark designed to analyze how VLM performance can be affected by structure type and structural density of visualizations depicting frequent patterns mined from sequence data. ProcVQA consists of mined process visualizations spanning three structure types (linear sequences, tree, graph) with varying levels of structural density (quantified using the number of nodes and edges), with expert-validated QA pairs on these visualizations. We evaluate 21 proprietary and open-source models on the dataset on two major tasks: visual data extraction (VDE) and visual question answering (VQA) (with four categories of questions). Our analysis reveals three key findings. First, models exhibit steep performance drops on multi-hop reasoning, with question type and structure type impacting the degradation. Second, structural density strongly affects VDE performance: hallucinations and extraction errors increase with edge density, even in frontier models. Third, extraction accuracy does not necessarily translate into strong reasoning ability. By isolating structural factors through controlled visualization generation, ProcVQA enables precise identification of VLM limitations. ProcVQA is available at: https://github.com/kzintas/ProcVQA.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAGuanhua Chen, Yutong Yao, Shenghe Sun, Ci-Jun Gao et al.ACL Findings 2026
  2. Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language ModelsIdo Cohen, Daniela Gottesman, Mor Geva, Raja GiryesACL 2025
  3. Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and RobustnessSrija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu et al.EMNLP Findings 2024 · 5 citations
  4. FeVisQA: Free-Form Question Answering over Data VisualizationsYuanfeng Song, Jinwei Lu, Yuanwei Song, Caleb Chen Cao et al.ICDE 2025 · 2 citations
  5. ChartR: Evaluating Reasoning Accuracy and Robustness in Chart Question AnsweringXiaojun Chen, Sixiao Luo, Ziqi Liu, Min Yang et al.CVPR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines