The Rational Agent Benchmark for Data Visualization
Yifan Wu, Ziyang Guo, Michalis Mamakos, Jason D. Hartline, Jessica Hullman
Abstract
Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Designing for Disclosure in Data Visualizations cites Visual Reasoning Strategies for Effect Size Judgments and Decisions
- Causal Support: Modeling Causal Inferences with Visualizations cites Visual Reasoning Strategies for Effect Size Judgments and Decisions
- Odds and Insights: Decision Quality in Exploratory Data Analysis Under Uncertainty cites Visual Reasoning Strategies for Effect Size Judgments and Decisions
- A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide? cites Visual Reasoning Strategies for Effect Size Judgments and Decisions
- EVM: Incorporating Model Checking into Exploratory Visual Analysis cites Causal Support: Modeling Causal Inferences with Visualizations
- EVM: Incorporating Model Checking into Exploratory Visual Analysis cites Bayesian-Assisted Inference from Visualized Data
- Decision Theoretic Foundations for Experiments Evaluating Human Decisions cites A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?
- Decision Theoretic Foundations for Experiments Evaluating Human Decisions cites Visual Reasoning Strategies for Effect Size Judgments and Decisions
- Decision Theoretic Foundations for Experiments Evaluating Human Decisions cites Bayesian-Assisted Inference from Visualized Data
Cited by top-tier papers7
- Odds and Insights: Decision Quality in Exploratory Data Analysis Under UncertaintyAbhraneel Sarma, Xiaoying Pu, Yuan Cui, Michael Correll et al.CHI 2024 · 8 citations
- Decision Theoretic Foundations for Experiments Evaluating Human DecisionsJessica Hullman, Alex Kale, Jason D. HartlineCHI 2025 · 6 citations
- Designing for Disclosure in Data VisualizationsKrisha Mehta, Gordon Kindlmann, Alex KaleIEEE VIS 2025 · 1 citation
- The Value of Information in Human-AI Decision-makingZiyang Guo, Yifan Wu, Jason D. Hartline, Jessica HullmanICLR 2026
- Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory AnalysisPriyanka Nanayakkara, Hyeok Kim, Yifan Wu, Ali Sarvghad et al.S&P 2024
Builds on6
- Visual Reasoning Strategies for Effect Size Judgments and DecisionsAlex Kale, Matthew Kay, Jessica HullmanIEEE VIS 2020 · 112 citations
- A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?Evanthia Dimara, John T. StaskoIEEE VIS 2021 · 56 citations
- Bayesian-Assisted Inference from Visualized DataYea-Seul Kim, Paula Kayongo, Madeleine Grunde-McLaughlin, Jessica HullmanIEEE VIS 2020 · 40 citations
- Causal Support: Modeling Causal Inferences with VisualizationsAlex Kale, Yifan Wu, Jessica HullmanIEEE VIS 2021 · 27 citations
- EVM: Incorporating Model Checking into Exploratory Visual AnalysisAlex Kale, Ziyang Guo, Xiaoli Qiao, Jeffrey Heer et al.IEEE VIS 2023 · 16 citations
Related papers
- Trust Your Gut: Comparing Human and Machine Inference from Noisy VisualizationsRatanond Koonchanok, Michael E. Papka, Khairi RedaIEEE VIS 2024
- Towards Modeling Visualization Processes as Dynamic Bayesian NetworksChristian HeineIEEE VIS 2020 · 7 citations
- Visualization EquilibriumPaula Kayongo, Glenn Sun, Jason D. Hartline, Jessica HullmanIEEE VIS 2021 · 3 citations
- Does Interaction Improve Bayesian Reasoning with Visualization?Abigail Mosca, Alvitta Ottley, Remco ChangCHI 2021 · 14 citations
- A Novel Lens on Metacognition in VisualizationMengyu Chen, Andrew Yang, Seungchan Min, Kristy A. Hamilton et al.CHI 2025 · 7 citations
