Lune

EMNLP Findings2024

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

Sihang Zhao, Youliang Yuan, Xiaoying Tang, Pinjia He

Abstract

Multimodal Large Language Models (MLLMs) demonstrate a strong understanding of the real world and can even handle complex tasks. However, they still fail on some straightforward visual question-answering (VQA) problems. This paper dives deeper into this issue, revealing that models tend to err when answering easy questions (e.g., Yes/No questions) about an image, even though they can correctly describe it. We refer to this model behavior discrepancy between difficult and simple questions as model laziness. To systematically investigate model laziness, we manually construct LazyBench, a benchmark that includes Yes/No, multiple choice, short answer questions, and image description tasks that are related to the same subjects in the images. Based on LazyBench, we observe that laziness widely exists in current advanced MLLMs (e.g., GPT-4o, Gemini-1.5-pro, Claude 3, LLaVA-1.5, LLaVA-1.6, and QWen-VL). We also analyzed the failure cases of LLaVA-1.5-13B on the VQA-v2 benchmark and discovered that about half of these failures are due to the model's laziness. This further highlights the importance of ensuring that the model fully utilizes its capability. To this end, we conduct a preliminary exploration of how to mitigate laziness and find that chain of thought can effectively avoid this issue. The data can be accessed at https://github.com/Akutagawa1998/LazyBench.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites Learning Transferable Visual Models From Natural Language Supervision
  • TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning cites Visual Instruction Tuning
  • Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. 3DCS: Datasets and Benchmark for Evaluating Conformational Sensitivity in Molecular RepresentationsXi Wang, Yang Zhang, Yingjia Zhang, Yejia Cai et al.ICLR 2026

Builds on12

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningFuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang et al.ICLR 2024 · 476 citations
  5. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question AnsweringYushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang et al.ICCV 2023 · 400 citations

Related papers

  1. VisNumBench: Evaluating Number Sense of Multimodal Large Language ModelsTengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong MingICCV 2025 · 1 citation
  2. Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language ModelsTianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian et al.CVPR 2024
  3. MuirBench: A Comprehensive Benchmark for Robust Multi-image UnderstandingFei Wang, Xingyu Fu, James Y. Huang, Zekun Li et al.ICLR 2025
  4. Visual Hallucinations of Multi-modal Large Language ModelsWen Huang, Hongbin Liu, Minxin Guo, Neil GongACL Findings 2024 · 30 citations
  5. AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language ModelsXiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang et al.EMNLP Findings 2024 · 18 citations
Dusk over the sea between two cliffs drawn in fine vertical lines