Discovering Clone Negatives via Adaptive Contrastive Learning for Image-Text Matching
Renjie Pan, Jihao Dong, Hua Yang
Abstract
In this paper, we identify a common yet challenging issue in image-text matching, i.e., clone negatives: negative image-text pairs that semantically resemble positive pairs, leading to ambiguous and sub-optimal matching outcomes. To tackle this issue, we propose Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters along with a modulating anchor to dynamically strengthen the compactness between positives and mitigate the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without the need for explicit training, allowing for progressive tuning and advanced in-batch supervision. Extensive experiments across several tasks demonstrate the effectiveness of AdaCL in image-text matching. Furthermore, we extend AdaCL to weakly-supervised image-text matching by replacing human-annotated descriptions with automatically generated captions, thereby increasing the number of potential clone negatives. AdaCL maintains robustness in this setting, alleviating the reliance on crowd-sourced annotations and laying a foundation for scalable vision-language contrastive learning. 𝐼 ! 𝐼 " A crowd of people in running outfits runs a marathon with two skyscrapers in background. 𝑇 ! 𝑇 " A group of people is running a marathon or race across the city.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Learning Transferable Visual Models From Natural Language Supervision cites A Simple Framework for Contrastive Learning of Visual Representations
- Learning Transferable Visual Models From Natural Language Supervision cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- OneLIP: Unlocking and Improving Long-Text Representations of CLIP via One-Stage Adaptation cites A Simple Framework for Contrastive Learning of Visual Representations
- OneLIP: Unlocking and Improving Long-Text Representations of CLIP via One-Stage Adaptation cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- OneLIP: Unlocking and Improving Long-Text Representations of CLIP via One-Stage Adaptation cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- OneLIP: Unlocking and Improving Long-Text Representations of CLIP via One-Stage Adaptation cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- OneLIP: Unlocking and Improving Long-Text Representations of CLIP via One-Stage Adaptation cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers2
Builds on32
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
Related papers
- Semantic-Augmented Image Clustering via Adaptive Multi-Modal CollaborationXiaohan Zhang, Chao Zhang, Deng Xu, Hong Yu et al.AAAI 2026
- Cross-modal Active Complementary Learning with Self-refining CorrespondenceYang Qin, Yuan Sun, Dezhong Peng, Joey Tianyi Zhou et al.NeurIPS 2023 · 49 citations
- Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language GroundingZhu Zhang, Zhou Zhao, Zhijie Lin, Jieming Zhu et al.NeurIPS 2020 · 74 citations
- MedCLIP: Contrastive Learning from Unpaired Medical Images and TextZifeng Wang, Zhenbang Wu, Dinesh Agarwal, Jimeng SunEMNLP 2022 · 907 citations
- Multi-Modal Representation Learning with Text-Driven Soft MasksJaeyoo Park, Bohyung HanCVPR 2023
