PAF: Prototype Adaptive Fusion for Test-Time Adaptation of Vision-Language Models

Leveraging Vision-Language Models (VLMs) like CLIP for various downstream tasks has emerged as a significant research trend. Recently, researchers have introduced Test-Time Adaptation (TTA) as a technique for models to learn online from unlabeled samples at test time, improving the generalization performance of VLMs to target domains. However, existing TTA methods either require expensive backpropagating gradient computations for each test sample or only extract knowledge from a limited number of historical test samples in the cache model, resulting in suboptimal adaptation performance. To address these limitations, we propose a Prototype Adaptive Fusion (PAF) framework, a novel TTA approach that makes full use of historical knowledge from test samples. Unlike traditional cache-based methods, which store only a few low-entropy samples per class, PAF introduces a prototype fusion mechanism that constructs class prototype representations through cumulatively merging features from qualified test samples. Furthermore, we propose an enhanced version, Easy-Hard PAF (EH-PAF), which adaptively applies a category-specific strategy based on CLIP prediction to improve performance. Extensive experiments across 15 diverse datasets demonstrate that our method consistently outperforms previous state-of-the-art approaches.