Lune

ACL Findings2024

ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, Tristan Thrush

Abstract

This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a captionimage pair, along with a "color-swapped" pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-ofdistribution word-order understanding task.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • Causal Graphical Models for Vision-Language Compositional Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Causal Graphical Models for Vision-Language Compositional Understanding cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • Causal Graphical Models for Vision-Language Compositional Understanding cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Causal Graphical Models for Vision-Language Compositional Understanding cites Learning Transferable Visual Models From Natural Language Supervision
  • Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Test-Time Matching: Unlocking Compositional Reasoning in Multimodal ModelsYinglun Zhu, Jiancheng Zhang, Fuzhi TangICLR 2026 · 5 citations
  2. Causal Graphical Models for Vision-Language Compositional UnderstandingFiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi et al.ICLR 2025
  3. Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal PerspectiveXiangru Zhu, Penglei Sun, Yaoxian Song, Yanghua Xiao et al.ICLR 2025

Builds on11

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations

Related papers

  1. Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMsYikang Zhou, Tao Zhang, Shilin Xu, Shihao Chen et al.ICCV 2025 · 2 citations
  2. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMsShengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma et al.CVPR 2024 · 111 citations
  3. Winoground: Probing Vision and Language Models for Visio-Linguistic CompositionalityTristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh et al.CVPR 2022 · 179 citations
  4. On Evaluating Adversarial Robustness of Large Vision-Language ModelsYunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang et al.NeurIPS 2023 · 404 citations
  5. Why are Visually-Grounded Language Models Bad at Image Classification?Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh et al.NeurIPS 2024 · 128 citations
Dusk over the sea between two cliffs drawn in fine vertical lines