Lune

ACM MM2024Top-tier venue

ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks

Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, Zhongyu Wei, Xuanjing Huang

Abstract

Recent years have witnessed remarkable progress in the development of large vision-language models (LVLMs). Benefiting from the strong language backbones and efficient cross-modal alignment strategies, LVLMs exhibit surprising capabilities to perceive visual signals and perform visually grounded reasoning. However, the capabilities of LVLMs have not been comprehensively and quantitatively evaluated. Most existing multi-modal benchmarks require task-oriented input-output formats, posing great challenges to automatically assess the freeform text output of LVLMs. To effectively leverage the annotations available in existing benchmarks and reduce the manual effort required for constructing new benchmarks, we propose to re-formulate existing benchmarks into unified LVLMcompatible formats. Through systematic data collection and reformulation, we present the ReForm-Eval benchmark, offering substantial data for evaluating various capabilities of LVLMs. Based on ReForm-Eval, we conduct extensive experiments, thoroughly analyze the strengths and weaknesses of existing LVLMs, and identify the underlying factors. Our benchmark and evaluation framework will be open-sourced as a cornerstone for advancing the development of LVLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-Modal Language Models cites Visual Instruction Tuning
  • Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-Modal Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-Modal Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-Modal Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs cites Visual Instruction Tuning
  • Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs cites Visual Instruction Tuning
  • AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Unifying Local and Global Knowledge: Empowering Large Language Models as Political Experts with Knowledge GraphsXinyi Mou, Zejun Li, Hanjia Lyu, Jiebo Luo et al.WWW 2024 · 20 citations
  2. Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-Modal Language ModelsShitian Zhao, Zhuowan Li, Yadong Lu, Alan L. Yuille et al.CVPR 2024 · 4 citations
  3. AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMsXuanwen Ding, Chengjun Pan, Zejun Li, Jiwen Zhang et al.ACL 2026 · 1 citation
  4. Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMsJie Zhang, Zhongqi Wang, Mengqi Lei, Zheng Yuan et al.ICLR 2025

Builds on23

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  5. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations

Related papers

  1. Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive EvaluationHong-Tao Yu, Yuxin Peng, Serge J. Belongie, Xiu-Shen WeiICLR 2026 · 21 citations
  2. Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem SolvingZixian Guo, Ming Liu, Qilong Wang, Zhilong Ji et al.ICCV 2025 · 1 citation
  3. CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language ModelsZihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei et al.AAAI 2025 · 36 citations
  4. Are Large Vision Language Models Good Game Players?Xinyu Wang, Bohan Zhuang, Qi WuICLR 2025
  5. SpaCE-Eval: A Benchmark for Real-World Multi-Modal ReasoningXuyou Yang, Yucheng Zhao, Wenxuan Zhang, Immanuel KohICLR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines