Lune

ACL Findings2025

Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward

Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, Yi R. Fung

Abstract

Large Vision Language Models (LVLMs) excel in various vision-language tasks. Yet, their robustness to visual variations in position, scale, orientation, and context that objects in natural scenes inevitably exhibit due to changes in viewpoint and environment remains largely underexplored. To bridge this gap, we introduce V 2 R-Bench, a comprehensive benchmark framework for evaluating Visual Variation Robustness of LVLMs, which encompasses automated evaluation dataset generation and principled metrics for thorough robustness assessment. Through extensive evaluation on 21 LVLMs, we reveal a surprising vulnerability to visual variations, in which even advanced models that excel at complex visionlanguage tasks significantly underperform on simple tasks such as object recognition. Interestingly, these models exhibit a distinct visual position bias that contradicts theories of effective receptive fields, and demonstrate a humanlike visual acuity threshold. To identify the source of these vulnerabilities, we present a systematic framework for component-level analysis, featuring a novel visualization approach for aligned visual features. Results show that these vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment. Complementary experiments with synthetic data further demonstrate that these limitations are fundamentally architectural deficiencies, scoring the need for architectural innovations in future LVLM designs. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models cites Visual Instruction Tuning
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language ModelsRohit Saxena, Alessandro Suglia, Pasquale MinerviniICML 2026 · 7 citations
  2. "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language ModelsKapil Garg, Xinru Tang, Jimin Heo, Dwayne R. Morgan et al.CHI 2026 · 2 citations
  3. MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence CalibrationZhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang et al.ACL 2025

Builds on24

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  5. Do Vision Transformers See Like Convolutional Neural Networks?Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang et al.NeurIPS 2021 · 1,553 citations

Related papers

  1. MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMsHuiyi Chen, Jiawei Peng, Dehai Min, Changchang Sun et al.ICML 2026 · 18 citations
  2. V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model InteractionYiming Zhao, Yu Zeng, Yukun Qi, YaoYang Liu et al.ICLR 2026 · 8 citations
  3. Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language UnderstandingWujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan et al.CVPR 2024 · 6 citations
  4. Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language ModelsMingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li et al.ICML 2024 · 32 citations
  5. Response Wide Shut? Surprising Observations in Basic Vision Language Model CapabilitiesShivam Chandhok, Wan-Cyuan Fan, Vered Shwartz, Vineeth N. Balasubramanian et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines