Beyond Single Images: A Comprehensive Benchmark for Album-Level Vision-Language Understanding
Abstract
Automatic album organization has been studied extensively over the past decades due to significant progress in digital photography. Recent Vision-Language Models (VLMs) have shown strong performance on multi-image understanding, making them natural candidates for automating album organization workflows. While VLMs' abilities in multi-image understanding have been widely studied, their performance on album organization remains underexplored. To bridge this gap, we introduce AlbumBench, the first comprehensive benchmark for automatic album organization. Specifically, we (1) define album organization tasks as photo selection for album-specific user objectives, photo rating according to how well user intents are fulfilled, and album-specific photo grouping given a user query that requires contextual understanding of the album; (2) establish AlbumBench, a benchmark dataset containing 27,051 images across 641 albums with 5 annotations per image; and (3) evaluate mainstream open-source and proprietary VLMs on AlbumBench. We show that AlbumBench presents unique challenges compared to traditional multi-image understanding benchmarks due to its requirement for understanding album context and user intent. Our findings reveal a significant performance gap between open-source and proprietary VLMs on album organization tasks. Despite this gap, even the best-performing proprietary models sometimes struggle with tasks that humans find relatively easy. We hope that AlbumBench can serve as a foundation for unifying album organization research and motivate improvements in VLMs' performance on these tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding cites MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites Learning Transferable Visual Models From Natural Language Supervision
Builds on5
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIXiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang et al.CVPR 2024 · 213 citations
- Context Length Alone Hurts LLM Performance Despite Perfect RetrievalYufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali et al.EMNLP Findings 2025 · 38 citations
- Improving Unsupervised Image Clustering With Robust LearningSungwon Park, Sungwon Han, Sundong Kim, Danu Kim et al.CVPR 2021
- MuirBench: A Comprehensive Benchmark for Robust Multi-image UnderstandingFei Wang, Xingyu Fu, James Y. Huang, Zekun Li et al.ICLR 2025
Related papers
- BannerBench: Benchmarking Vision Language Models for Multi-Ad Selection with Human PreferencesHiroto Otake, Peinan Zhang, Yusuke Sakai, Masato Mita et al.EMNLP Findings 2025
- RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world ScenariosFei Zhao, Chengqiang Lu, Yufan Shen, Qimeng Wang et al.EMNLP Findings 2025
- OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language ModelsQiguang Chen, Chengyu Luan, Jiajun Wu, Qiming Yu et al.ACL 2026 · 1 citation
- VL-RouterBench: A Benchmark for Vision–Language Model RoutingZhehao Huang, Baijiong Lin, Jingyuan Zhang, Jingying Wang et al.CVPR 2026 · 6 citations
- STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language ModelsMahiro Ukai, Shuhei Kurita, Nakamasa InoueACM MM 2025
