Lune

ACL Findings2024

CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples

Jianrui Zhang, Mu Cai, Tengyang Xie, Yong Jae Lee

Abstract

We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In particular, we identify two critical underexplored problems: the neglect of physically grounded reasoning (counting and position understanding) and the potential of using highly capable text and image generation models for semantic counterfactual fine-tuning. Our work pioneers an approach in addressing these gaps.

We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning. We then apply simple data augmentation using the grounded image generation model GLIGEN to generate fine-tuning data, resulting in significant performance improvements: +33% and +37% for CLIP and LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, we exploit the capabilities of high-performing text generation and image generation models, specifically GPT-4V and DALLE-3, to curate challenging semantic counterfactuals, thereby further enhancing compositional reasoning capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms GPT-4V.

To facilitate future research, we release our code, dataset, benchmark, and checkpoints at https://countercurate.github.io/.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

15 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs cites Visual Instruction Tuning
  • Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs cites Flamingo: a Visual Language Model for Few-Shot Learning
  • SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models cites Visual Instruction Tuning
  • SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • VisMin: Visual Minimal-Change Understanding cites Visual Instruction Tuning
  • VisMin: Visual Minimal-Change Understanding cites Learning Transferable Visual Models From Natural Language Supervision
  • VisMin: Visual Minimal-Change Understanding cites Teaching CLIP to Count to Ten
  • Teaching CLIP to Count to Ten cites Learning Transferable Visual Models From Natural Language Supervision
  • Teaching CLIP to Count to Ten cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning cites SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
  • SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning cites Visual Instruction Tuning
  • SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning cites Learning Transferable Visual Models From Natural Language Supervision
  • SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Object-centric binding in Contrastive Language-Image Pretraining cites Visual Instruction Tuning
  • Object-centric binding in Contrastive Language-Image Pretraining cites VisMin: Visual Minimal-Change Understanding
  • Object-centric binding in Contrastive Language-Image Pretraining cites Learning Transferable Visual Models From Natural Language Supervision
  • Object-centric binding in Contrastive Language-Image Pretraining cites Teaching CLIP to Count to Ten
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers15

  1. SpatialRGPT: Grounded Spatial Reasoning in Vision-Language ModelsAn-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo et al.NeurIPS 2024 · 412 citations
  2. Prism: A Framework for Decoupling and Assessing the Capabilities of VLMsYuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang et al.NeurIPS 2024 · 49 citations
  3. SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial ReasoningYang Liu, Ming Ma, Xiaomin Yu, Pengxiang Ding et al.NeurIPS 2025 · 46 citations
  4. VisMin: Visual Minimal-Change UnderstandingRabiul Awal, Saba Ahmadi, Le Zhang, Aishwarya AgrawalNeurIPS 2024 · 25 citations
  5. Object-centric binding in Contrastive Language-Image PretrainingRim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal et al.NeurIPS 2025 · 14 citations

Builds on13

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  5. Teaching CLIP to Count to TenRoni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada et al.ICCV 2023 · 196 citations

Related papers

  1. Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional UnderstandingLe Zhang, Rabiul Awal, Aishwarya AgrawalCVPR 2024 · 7 citations
  2. CR³: Boosting Compositional Reasoning in MLLMs Through Rule-Based Reinforcement LearningShun Qian, Bingquan Liu, Chengjie Sun, Peijin Xie et al.AAAI 2026
  3. Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision EncoderSiting Li, Pang Wei Koh, Simon Shaolei DuACL 2025
  4. CF-VLM: CounterFactual Vision-Language Fine-tuningJusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang et al.NeurIPS 2025 · 71 citations
  5. Advancing Compositional Awareness in CLIP with Efficient Fine-TuningAmit Peleg, Naman Deep Singh, Matthias HeinNeurIPS 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines