Lune

EMNLP Findings2024

CONSTRUCTURE: Benchmarking CONcept STRUCTUre REasoning for Multimodal Large Language Models

Zhiwei Zha, Xiangru Zhu, Yuanyi Xu, Chenghua Huang, Jingping Liu, Zhixu Li, Xuwu Wang, Yanghua Xiao, Bei Yang, Xiaoxiao Xu

Abstract

Multimodal Large Language Models (MLLMs) have shown promising results in various tasks, but their ability to perceive the visual world with deep, hierarchical understanding similar to humans remains uncertain. To address this gap, we introduce CONSTRUCTURE, a novel concept-level benchmark to assess MLLMs' hierarchical concept understanding and reasoning abilities. Our goal is to evaluate MLLMs across four key aspects: 1) Understanding atomic concepts at different levels of abstraction; 2) Performing upward abstraction reasoning across concepts; 3) Achieving downward concretization reasoning across concepts; and 4) Conducting multi-hop reasoning between sibling or common ancestor concepts. Our findings indicate that even state-of-the-art multimodal models struggle with concept structure reasoning (e.g., GPT-4o averages a score of 62.1%). We summarize key findings of MLLMs in concept structure reasoning evaluation. Morever, we provide key insights from experiments using CoT prompting and fine-tuning to enhance their abilities.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Visual Instruction Tuning
  • MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  3. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  4. MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
  5. Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng et al.ICCV 2019 · 1,018 citations

Related papers

  1. Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMsYikang Zhou, Tao Zhang, Shilin Xu, Shihao Chen et al.ICCV 2025 · 2 citations
  2. MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and EfficiencyDongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li et al.ICML 2025
  3. When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-ThoughtYiyang Zhou, Haoqin Tu, Zijun Wang, Zeyu Wang et al.CVPR 2026 · 17 citations
  4. M³CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-ThoughtQiguang Chen, Libo Qin, Jin Zhang, Zhi Chen et al.ACL 2024 · 13 citations
  5. AnchorCoT: Anchors Pave the Way for Multi-hop ReasoningTianshi Ming, Xian Wu, Yingying Zhang, Zichuan Fu et al.ACL Findings 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines