ACL Findings2023
Delving into the Openness of CLIP
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, Xu Sun
Abstract
Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descriptions instead of discrete category IDs. This allows for open-vocabulary visual recognition, where the model can recognize images from an open class set (also known as an open vocabulary) in a zero-shot manner. However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice. To address this, we resort to an incremental perspective to assess the openness through vocabulary expansions, and define extensibility to measure a model's ability to handle novel classes. Our evaluation shows that CLIP-like models are not truly open, and their performance deteriorates as the vocabulary expands. We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity. Our investigation reveals that the overestimation of openness is due to confusion among competing text features, rather than a failure to capture the similarity between image features and text features of novel classes. We hope that our investigation and analysis will facilitate future research on the CLIP openness issue. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Learning Transferable Visual Models From Natural Language Supervision
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding cites Learning Transferable Visual Models From Natural Language Supervision
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation cites Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation cites Learning Transferable Visual Models From Natural Language Supervision
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation cites Flamingo: a Visual Language Model for Few-Shot Learning
- Vocabulary Scaling Law: Tuning Open-vocabulary Predictors for Their Openness cites Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Vocabulary Scaling Law: Tuning Open-vocabulary Predictors for Their Openness cites Learning Transferable Visual Models From Natural Language Supervision
- Vocabulary Scaling Law: Tuning Open-vocabulary Predictors for Their Openness cites Flamingo: a Visual Language Model for Few-Shot Learning
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- Can Language Models Understand Physical Concepts? cites Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Can Language Models Understand Physical Concepts? cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Can Language Models Understand Physical Concepts? cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Flamingo: a Visual Language Model for Few-Shot Learning cites Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Flamingo: a Visual Language Model for Few-Shot Learning cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video UnderstandingShuhuai Ren, Linli Yao, Shicheng Li, Xu Sun et al.CVPR 2024 · 83 citations
- Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual RecognitionShuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang et al.NeurIPS 2023 · 46 citations
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and TransportationYuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu et al.NeurIPS 2024 · 45 citations
- Can Language Models Understand Physical Concepts?Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng et al.EMNLP 2023 · 13 citations
- Vocabulary Scaling Law: Tuning Open-vocabulary Predictors for Their OpennessZiliang Chen, Yulu Li, Liangda Fang, jusheng zhang et al.CVPR 2026
Builds on14
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
- Align before Fuse: Vision and Language Representation Learning with Momentum DistillationJunnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty et al.NeurIPS 2021 · 2,985 citations
Related papers
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual KnowledgeFawaz Sammani, Nikos DeligiannisNeurIPS 2024 · 15 citations
- RegionCLIP: Region-based Language-Image PretrainingYiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li et al.CVPR 2022 · 481 citations
- Non-Contrastive Learning Meets Language-Image Pre-TrainingJinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang et al.CVPR 2023
- Demystifying CLIP DataHu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang et al.ICLR 2024 · 249 citations
- CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallyDarina Koishigarina, Arnas Uselis, Seong Joon OhICLR 2026 · 33 citations
