Lune

EMNLP Findings2023

ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories

Heming Xia, Qingxiu Dong, Lei Li, Jingjing Xu, Tianyu Liu, Ziwei Qin, Zhifang Sui

Abstract

Recently, Large Language Models (LLMs) have been serving as general-purpose interfaces, posing a significant demand for comprehensive visual knowledge. However, it remains unclear how well current LLMs and their visually augmented counterparts (VaLMs) can master visual commonsense knowledge. To investigate this, we propose IMAGENETVC, a human-annotated dataset specifically designed for zero-and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing IMAGENETVC, we benchmark the fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Furthermore, we analyze the factors affecting the visual commonsense knowledge of large-scale models, providing insights into the development of language models enriched with visual commonsense knowledge. Our code and dataset are available at https://github.com/ hemingkx/ImageNetVC.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks cites Flamingo: a Visual Language Model for Few-Shot Learning
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Calibrate Before Use: Improving Few-shot Performance of Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented BenchmarksZejun Li, Ye Wang, Mengfei Du, Qingwen Liu et al.ACM MM 2024 · 1 citation

Builds on14

  1. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations

Related papers

  1. A Systematic Investigation of Commonsense Knowledge in Large Language ModelsXiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d'Autume et al.EMNLP 2022 · 34 citations
  2. Zero-shot Commonsense Reasoning over Machine ImaginationHyuntae Park, Yeachan Kim, Jun-Hyung Park, SangKeun LeeEMNLP Findings 2024 · 1 citation
  3. VCD: A Dataset for Visual Commonsense Discovery in ImagesXiangqing Shen, Fanfan Wang, Siwei Wu, Rui XiaACL Findings 2025 · 2 citations
  4. Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language ModelsBoyu Jia, Junzhe Zhang, Huixuan Zhang, Xiaojun WanEMNLP Findings 2025
  5. Visually-Augmented Language ModelingWeizhi Wang, Li Dong, Hao Cheng, Haoyu Song et al.ICLR 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines