Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding
Jaehyun Jeon, Min Soo Kim, Janghan Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu
Abstract
User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent studies have explored UI evaluation using Multimodal Large Language Models (MLLMs), they largely focus on surface-level features, overlooking how design choices influence user behavior at scale. To fill this gap, we introduce WiserUI-Bench, a novel benchmark for multimodal understanding of how UI/UX design affects user behavior, built on 300 real-world UI image pairs from industry A/B tests, with empirically validated winners that induced more user actions. For future design progress in practice, post-hoc understanding of why such winners succeed with mass users is also required; we support this via expert-curated key interpretations for each instance. Experiments across multiple MLLMs on WiserUI-Bench for two main tasks, (1) predicting the more effective UI image between an A/B-tested pair, and (2) explaining it post-hoc in alignment with expert interpretations, show that models exhibit limited understanding of the behavioral impact of UI/UX design. We believe our work will foster research on leveraging MLLMs for visual design in user behavior contexts.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext f9eeec3e-6955-4563-8061-b7546aa46600Cited by top-tier papers1
Ask how each one uses itBuilds on15
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
Related papers
- Leveraging Multimodal LLM for Inspirational User Interface SearchSeokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim et al.CHI 2025 · 10 citations
- CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface DesignDaeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim et al.AAAI 2026
- Component-based Reusable UI Code Generation for Complex Websites via Semantic Segmentation and Fine-grained FeedbackJingyu Xiao, Jiantong Qin, Shuoqi Li, Man Ho Lam et al.KDD 2026 · 1 citation
- Tiny Scales, Great Challenges: The Limits of Multimodal LLMs in Scale RecognitionJihang Jin, Ronghao Chen, Hao Zhang, Ziyan Liu et al.ACL 2026
- SO-Bench: A Structural Output Evaluation of Multimodal LLMDi Feng, Kaixin Ma, Feng Nan, Haofeng Chen et al.CVPR 2026
