MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for lowresource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages-over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multiimage tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs. * Equal contribution. Images-To-Sentences (I2S) Which sentence best matches the topic of the images? The images and the sentences each belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel". Choose one sentence from A, B, C, or D. Output only a single letter! # Images # Sentences A. Maroochydore führte am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweitem.``B . Es wurden keine schwere Verletzungen gemeldet, jedoch mussten mindestens fünf der zur Zeit der Explosion Anwesenden aufgrund von Schocksymptomen behandelt werden.C . ```Finnland ist ein großartiges Reiseziel für Bootstouren. Das "Land der tausend Seen" hat auch Tausende von Inseln -in den Seen und in den Küstenarchipelen.D . ```Es ist auch nicht erforderlich, dass Sie eine lokale Nummer von der Gemeinde erhalten, in der Sie leben. Sie können eine Internetverbindung über Satellit in der Wildnis v on Chicken in Alaska erhalten und eine Nummer auswählen, die vorgibt, dass Sie im sonnigen Arizona sind.``Ỳ our answer letter: