Lune

ACL Findings2024

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, Nanyun Peng

Abstract

Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose a large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric (Rohrbach et al., 2018) and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human-annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding cites Flamingo: a Visual Language Model for Few-Shot Learning
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • HalLoc: Token-level Localization of Hallucinations for Vision Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • HalLoc: Token-level Localization of Hallucinations for Vision Language Models cites Visual Instruction Tuning
  • HalLoc: Token-level Localization of Hallucinations for Vision Language Models cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • SCIEval: Evaluating and Benchmarking the Faithfulness of Scientific Image Generation and Interpretation with Large Multimodal Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations cites Visual Instruction Tuning
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and RecommendationsMd. Tahmid Rahman Laskar, Sawsan Alqahtani, M. Saiful Bari, Mizanur Rahman et al.EMNLP 2024 · 47 citations
  2. When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and UnderstandingYan Shu, Hangui Lin, Yexin Liu, Yan Zhang et al.NeurIPS 2025 · 17 citations
  3. SafeWorld: Geo-Diverse Safety AlignmentDa Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang et al.NeurIPS 2024 · 14 citations
  4. HalLoc: Token-level Localization of Hallucinations for Vision Language ModelsEunkyu Park, Minyeong Kim, Gunhee KimCVPR 2025
  5. SCIEval: Evaluating and Benchmarking the Faithfulness of Scientific Image Generation and Interpretation with Large Multimodal ModelsGuanghui Ye, Huan Zhao, Zhixue Zhao, Tengfei Ma et al.CVPR 2026

Builds on14

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  5. ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations

Related papers

  1. FIHA: Automated Fine-grained Hallucinations Evaluations in Large Vision Language Models with Davidson Scene GraphsBowen Yan, Zhengsong Zhang, Liqiang Jing, Eftekhar Hossain et al.ACL Findings 2025 · 2 citations
  2. THRONE: An Object-Based Hallucination Benchmark for the Free-Form Generations of Large Vision-Language ModelsPrannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler et al.CVPR 2024
  3. Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language ModelsChaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye et al.ACM MM 2024 · 19 citations
  4. Beyond Facts: Evaluating Intent Hallucination in Large Language ModelsYijie Hao, Haofei Yu, Jiaxuan YouACL 2025
  5. VisDiaHalBench: A Visual Dialogue Benchmark For Diagnosing Hallucination in Large Vision-Language ModelsQingxing Cao, Junhao Cheng, Xiaodan Liang, Liang LinACL 2024 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines