Lune

EMNLP Findings2025

BannerBench: Benchmarking Vision Language Models for Multi-Ad Selection with Human Preferences

Hiroto Otake, Peinan Zhang, Yusuke Sakai, Masato Mita, Hiroki Ouchi, Taro Watanabe

Abstract

Web banner advertisements, which are placed on websites to guide users to a targeted landing page (LP), are still often selected manually because human preferences are important in selecting which ads to deliver. To automate this process, we propose a new benchmark, BannerBench, to evaluate the human preference-driven banner selection process using vision-language models (VLMs). This benchmark assesses the degree of alignment with human preferences in two tasks: a ranking task and a best-choice task, both using sets of five images derived from a single LP. Our experiments show that VLMs are moderately correlated with human preferences on the ranking task. In the best-choice task, most VLMs perform close to chance level across various prompting strategies. These findings suggest that although VLMs have a basic understanding of human preferences, most of them struggle to pinpoint a single suitable option from many candidates. Our benchmark dataset is available at https://huggingface. co/datasets/cyberagent/BannerBench.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language BenchmarkDongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang et al.ICML 2024 · 345 citations
  2. tinyBenchmarks: evaluating LLMs with fewer examplesFelipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun et al.ICML 2024 · 212 citations
  3. Automated Creative Optimization for E-Commerce AdvertisingJin Chen, Ju Xu, Gangwei Jiang, Tiezheng Ge et al.WWW 2021 · 22 citations
  4. Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask LearningHuaizheng Zhang, Yong Luo, Qiming Ai, Yonggang Wen et al.ACM MM 2020 · 17 citations
  5. CTR-Driven Advertising Image Generation with Multimodal Large Language ModelsXingye Chen, Wei Feng, Zhenbang Du, Weizhen Wang et al.WWW 2025 · 15 citations

Related papers

  1. Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric PerspectivesShaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima et al.ICCV 2025 · 25 citations
  2. Do MLLMs Exhibit Human-like Perceptual Behaviors? HVSBench: A Benchmark for MLLM Alignment with Human Perceptual BehaviorJiaying Lin, Shuquan Ye, Dan Xu, Wanli Ouyang, Rynson W. H. LauCVPR Findings 2026
  3. Beyond Single Images: A Comprehensive Benchmark for Album-Level Vision-Language UnderstandingShawn Huang, Brian L. Price, Yifei Fan, Bryan S. MorseCVPR 2026
  4. VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent EnvironmentsZelai Xu, Zhexuan Xu, Xiangmin Yi, Huining Yuan et al.CVPR 2026 · 3 citations
  5. ProgressLM: Towards Progress Reasoning in Vision-Language ModelsJianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu et al.ACL 2026 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines