CVPR Findings2026
Dual-Modality Anchor-Guided Filtering for Test-Time Prompt Tuning
Jungwon Choi, Eunwoo Kim
Abstract
Test-Time Prompt Tuning (TPT) adapts vision-language models using augmented views, but its effectiveness is hindered by the challenge of determining which views are beneficial. Standard entropy-based filtering relies on the internal confidence scores of the model, which are often miscalibrated under distribution shift, assigning high confidence to irrelevant crops or background regions while ignoring semantic content. To address this, we propose a dual-modality anchor-guided framework that grounds view selection in semantic evidence. We introduce a text anchor from attributerich descriptions, to provide fine-grained class semantics, and an adaptive image anchor that captures evolving testtime statistics. Using these anchors, we filter views based on alignment and confidence, ensuring that only informative views guide adaptation. Moreover, we treat the anchors as auxiliary predictive heads and combine their predictions with the original output in a confidence-weighted ensemble, yielding a stable supervision signal for prompt updates. Extensive experiments on 15 benchmark datasets demonstrate new state-of-the-art performance, highlighting the contribution of anchor-guided supervision as a foundation for robust prompt updates.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- Sigmoid Loss for Language Image Pre-Training cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Builds on25
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations
Related papers
- Doubly Debiased Test-Time Prompt Tuning for Vision-Language ModelsFei Song, Yi Li, Rui Wang, Jiahuan Zhou et al.AAAI 2026
- Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language ModelsManli Shu, Weili Nie, De-An Huang, Zhiding Yu et al.NeurIPS 2022 · 603 citations
- Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language ModelsJia-Wei Hai, Yijun Wang, Xiu-Shen WeiICML 2026
- USE: A Unified Self-Ensembling Framework for Test-Time Prompt TuningSiru Jiang, Jian Liang, Ran He, Tieniu TanICML 2026
- Diverse Data Augmentation with Diffusions for Effective Test-time Prompt TuningChun-Mei Feng, Kai Yu, Yong Liu, Salman Khan et al.ICCV 2023 · 172 citations
