VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models

We introduce VLURes, a multilingual benchmark for evaluating Vision-Language Models (VLMs) under long-text grounding: selecting and reasoning over the image-relevant subset of article-length text that contains distractors and ungrounded claims. VLURes contains 4,000 web-curated image+long-text pairs across English (En), Japanese (Ja), Swahili (Sw), and Urdu (Ur) and 10 topical categories, and defines eight tasks spanning image-only perception (OR, SU, RU, SS, IC) and image+text grounding (ITM, Unrelatedness, VQA). To construct webrealistic pairs, we apply language-adapted CLIP alignment to select representative images and filter weakly grounded pages. Across 10 proprietary and open VLMs evaluated under zero-shot and one-shot prompting, with and without rationales, the best model (GPT-4o) reaches 90.8% overall accuracy but remains 6.7 points below human performance (97.5%) on Object Recognition, and cross-lingual sensitivity persists, while open models are substantially weaker and often lack reliable multilingual VL support. VLURes provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. Dataset trustworthiness and intrinsic difficulty. Web pages often contain multiple images, icons, and off-topic media. To pair articles with representative images while reducing spurious matches, we use a language-adapted CLIP alignment step that selects the most relevant image and filters weakly grounded pages. We then characterize intrinsic sample difficulty using (i) text length, (ii) the adapted alignment score, and (iii) an ambiguity margin between the best and runner-up images. These signals help quantify how web-realistic noise and language shift interact with model behavior, complementing prior findings on multilingual generalization gaps (Pfeiffer et al., 2022; Geigle et al., 2024; Chen et al., 2023b). Summary of findings and contributions. Across 10 VLMs and four prompting regimes (zero-shot vs. one-shot, with vs. without rationales), one-shot prompting and rationales often improve absolute accuracy, but do not reliably close cross-lingual gaps. Long-text grounding tasks are consistently harder than image-only tasks, especially under language shift. Open models face a dual barrier: weaker performance and unreliable multilingual VL support, even when their base large language models (LLMs) are multilingual (Geigle et al., 2024; Chen et al., 2023b). These trends align with evidence that prompting can change apparent reasoning without guaranteeing grounded behavior (Wei et al., 2022b; Turpin et al., 2023). Overall, VLURes contributes: (i) a multilingual image+long-text benchmark with realistic web provenance, (ii) an eight-task suite that cleanly separates image-only competence from long-text grounding (including Unrelatedness), and (iii) a standardized evaluation of proprietary and open VLMs under controlled prompting regimes to quantify multilingual robustness. Related Work Multimodal benchmarks beyond captions. A large fraction of VLM evaluation has historically focused on captioning and VQA-style supervision with short inputs, including MSCOCO (Lin et al., 2015), Flickr30k (Plummer et al., 2016), VQAv2 (Goyal et al., 2017a), GQA (Hudson and Manning, 2019), and TextVQA (Singh et al., 2019). More recent benchmarks (Table 2 ) aim to stress broader capabilities via instruction-style prompts or multi-domain reasoning, such as MME (Fu et al., 2023), MMMU (Yue et al., 2023), MMBench (Liu et al., 2023d), SEED-Bench (Li et al., 2023a), and MM-Vet (Yu et al., 2023