Lune

ACL Findings2026

VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models

Jesse Atuhurra, Iqra Ali, Tomoya Iwakura, Hidetaka Kamigaito, Tatsuya Hiraoka

Abstract

We introduce VLURes, a multilingual benchmark for evaluating Vision-Language Models (VLMs) under long-text grounding: selecting and reasoning over the image-relevant subset of article-length text that contains distractors and ungrounded claims.VLURes contains 4,000 web-curated image+long-text pairs across English (En), Japanese (Ja), Swahili (Sw), and Urdu (Ur) and 10 topical categories, and defines eight tasks spanning image-only perception (OR, SU, RU, SS, IC) and image+text grounding (ITM, Unrelatedness, VQA).To construct webrealistic pairs, we apply language-adapted CLIP alignment to select representative images and filter weakly grounded pages.Across 10 proprietary and open VLMs evaluated under zero-shot and one-shot prompting, with and without rationales, the best model (GPT-4o) reaches 90.8% overall accuracy but remains 6.7 points below human performance (97.5%) on Object Recognition, and cross-lingual sensitivity persists, while open models are substantially weaker and often lack reliable multilingual VL support.VLURes provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. Dataset trustworthiness and intrinsic difficulty.Web pages often contain multiple images, icons, and off-topic media.To pair articles with representative images while reducing spurious matches, we use a language-adapted CLIP alignment step that selects the most relevant image and filters weakly grounded pages.We then characterize intrinsic sample difficulty using (i) text length, (ii) the adapted alignment score, and (iii) an ambiguity margin between the best and runner-up images.These signals help quantify how web-realistic noise and language shift interact with model behavior, complementing prior findings on multilingual generalization gaps (Pfeiffer et al., 2022; Geigle et al., 2024; Chen et al., 2023b). Summary of findings and contributions.Across 10 VLMs and four prompting regimes (zero-shot vs. one-shot, with vs. without rationales), one-shot prompting and rationales often improve absolute accuracy, but do not reliably close cross-lingual gaps.Long-text grounding tasks are consistently harder than image-only tasks, especially under language shift.Open models face a dual barrier: weaker performance and unreliable multilingual VL support, even when their base large language models (LLMs) are multilingual (Geigle et al., 2024; Chen et al., 2023b).These trends align with evidence that prompting can change apparent reasoning without guaranteeing grounded behavior (Wei et al., 2022b; Turpin et al., 2023).Overall, VLURes contributes: (i) a multilingual image+long-text benchmark with realistic web provenance, (ii) an eight-task suite that cleanly separates image-only competence from long-text grounding (including Unrelatedness), and (iii) a standardized evaluation of proprietary and open VLMs under controlled prompting regimes to quantify multilingual robustness. 2 Related Work Multimodal benchmarks beyond captions.A large fraction of VLM evaluation has historically focused on captioning and VQA-style supervision with short inputs, including MSCOCO (Lin et al., 2015), Flickr30k (Plummer et al., 2016), VQAv2 (Goyal et al., 2017a), GQA (Hudson and Manning, 2019), and TextVQA (Singh et al., 2019).More recent benchmarks (Table 2) aim to stress broader capabilities via instruction-style prompts or multi-domain reasoning, such as MME (Fu et al., 2023), MMMU (Yue et al., 2023), MMBench (Liu et al., 2023d), SEED-Bench (Li et al., 2023a), and MM-Vet (Yu et al., 2023).These resources have Model

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on3

  1. Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
  2. nocaps: novel object captioning at scaleHarsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang et al.ICCV 2019 · 631 citations
  3. IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and LanguagesEmanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy et al.ICML 2022 · 71 citations

Related papers

  1. VL-ICL Bench: The Devil in the Details of Multimodal In-Context LearningYongshuo Zong, Ondrej Bohdal, Timothy M. HospedalesICLR 2025
  2. VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual CuesJianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang et al.ACL 2025
  3. From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language ModelsMehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang et al.EMNLP 2024 · 6 citations
  4. Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric PerspectivesShaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima et al.ICCV 2025 · 25 citations
  5. VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMsBrigitta Malagurski Törtei, Yasser Dahou, Ngoc Dung Huynh, Wamiq Reyaz Para et al.CVPR 2026 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines