Semi-Supervised CLIP Adaptation by Enforcing Semantic and Trapezoidal Consistency
Abstract
Vision-language pre-training models, such as CLIP, have demonstrated strong capability in rapidly adapting to downstream tasks through fine-tuning, and have been widely applied across various tasks. However, when the downstream tasks are constrained by limited image-text paired data, CLIP struggles to effectively address the domain gap between the pre-training and the target tasks. To address this limitation, we propose a novel semi-supervised CLIP adaptation method coined SEMICLIP that leverages a small amount of image-text pairs alongside a large volume of images without text descriptions to enhance CLIP's cross-modal alignment. To effectively utilize unlabeled images, we introduce semantic concept mining to improve task-specific visual representations by matching images with relevant concepts mined from labeled data. Leveraging matched semantic concepts, we construct learnable surrogate captions for unlabeled images and optimize a trapezoidal consistency to regulate the geometric structure of image-text pairs in the representation space. Experimental results demonstrate that our approach significantly improves the adaptability of CLIP in target tasks with limited labeled data, achieving gains ranging from 1.72% -6.58% for zero-shot classification accuracy and 2.32% -3.23% for image-text retrieval performance on standard benchmarks. The source code is available at https://github.com/Gank0078/SemiCLIP.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Learning Transferable Visual Models From Natural Language Supervision
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers3
- Integrating Task-Specific and Universal Adapters for Pre-Trained Model-Based Class-Incremental LearningYan Wang, Da-Wei Zhou, Han-Jia YeICCV 2025 · 7 citations
- Robust Remote Sensing Image–Text Retrieval with Noisy Correspondenceqiya song, Yiqiang Xie, Yuan Sun, Renwei Dian et al.CVPR 2026 · 1 citation
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual RecognitionZheda Mai, Ping Zhang, Cheng-Hao Tu, Hong-You Chen et al.CVPR 2025
Builds on35
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
- RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
- Align before Fuse: Vision and Language Representation Learning with Momentum DistillationJunnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty et al.NeurIPS 2021 · 2,985 citations
Related papers
- S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist CaptionsSangwoo Mo, Minkyu Kim, Kyungmin Lee, Jinwoo ShinNeurIPS 2023 · 53 citations
- PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model PretrainingYuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang et al.NeurIPS 2022 · 168 citations
- Unsupervised Domain Adaptation for Referring Semantic SegmentationHaonan Shi, Wenwen Pan, Zhou Zhao, Mingmin Zhang et al.ACM MM 2023 · 5 citations
- SoftCLIP: Softer Cross-Modal Alignment Makes CLIP StrongerYuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu et al.AAAI 2024 · 80 citations
- RankCLIP: Ranking-Consistent Language-Image PretrainingYiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zhili Feng et al.ICCV 2025 · 1 citation
