Lune

ICLR2025Top-tier venue

Mitigating Spurious Correlations in Zero-Shot Multimodal Models

Shenyu Lu, Junyi Chai, Xiaoqian Wang

Abstract

Multimodal models or Vision Language Models (VLMs) have reshaped the paradigm in machine learning, offering zero-shot capabilities that require no additional training when adapted to new classification tasks. However, despite their advancements, spurious correlations still exist in VLMs. Existing approaches to tackle this issue often require target label annotations, contradicting the principle of zero-shot classification, or they primarily focus on a single modality, risking misalignment between text and image modalities. Others rely on extensive domain knowledge or large language models (LLMs) to characterize spurious features, making the performance sensitive to the generated prompts and undermining zero-shot capability. In response, we propose a new solution that tackles spurious correlations in VLMs within the zero-shot setting. Our approach utilizes a translation operation that preserves the latent space distribution to address issues of spurious correlations. In particular, our method is grounded in and inspired by a theoretical analysis, which identifies that the optimal translation directions are along the spurious vector. As VLMs unify two modalities, we compute spurious vectors from the text prompts and guide the translation for image embeddings, aligning the requirements for the fusion of different modalities in VLMs. We conducted experiments on benchmark datasets, which have shown significant improvements in worst-group accuracy. Additionally, our visualizations of VLMs further demonstrate the effectiveness of this intervention.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Label-Free Mitigation of Spurious Correlations in VLMs using Sparse Autoencoders cites Learning Transferable Visual Models From Natural Language Supervision
  • Label-Free Mitigation of Spurious Correlations in VLMs using Sparse Autoencoders cites WILDS: A Benchmark of in-the-Wild Distribution Shifts
  • SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias cites Learning Transferable Visual Models From Natural Language Supervision
  • Think Twice: Test-Time Reasoning for Robust CLIP Zero-Shot Classification cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Think Twice: Test-Time Reasoning for Robust CLIP Zero-Shot Classification cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach cites Learning Transferable Visual Models From Natural Language Supervision
  • Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach cites WILDS: A Benchmark of in-the-Wild Distribution Shifts
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Policy Contrastive Decoding for Robotic Foundation ModelsShihan Wu, Xu Luo, Ji Zhang, Junlin Xie et al.ICLR 2026 · 12 citations
  2. Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance ApproachAishwarya Agarwal, Srikrishna Karanam, Vineet GandhiCVPR 2026 · 2 citations
  3. Think Twice: Test-Time Reasoning for Robust CLIP Zero-Shot ClassificationShenyu Lu, Zhaoying Pan, Xiaoqian WangICCV 2025 · 1 citation
  4. Label-Free Mitigation of Spurious Correlations in VLMs using Sparse AutoencodersBharat Chandra Yalavarthi, Nalini K. Ratha, Venu GovindarajuICLR 2026
  5. SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal BiasWenqian Ye, Di Wang, Guangtao Zheng, Bohan Liu et al.AAAI 2026

Builds on27

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  5. WILDS: A Benchmark of in-the-Wild Distribution ShiftsPang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie et al.ICML 2021 · 1,773 citations

Related papers

  1. RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language ModelsMaya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari et al.NeurIPS 2024 · 28 citations
  2. Density-Aware Translation of Spurious Correlations in Zero-Shot VLMsAfsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah ErfaniICML 2026
  3. Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image RecognitionYicheng Liu, Jie Wen, Chengliang Liu, Xiaozhao Fang et al.ICML 2024 · 7 citations
  4. Bridge the Modality and Capability Gaps in Vision-Language Model SelectionChao Yi, Yuhang He, De-Chuan Zhan, Han-Jia YeNeurIPS 2024 · 32 citations
  5. Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?Hengyi Feng, Zeang Sheng, Meiyi Qiang, Yang Li et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines