Joint Test-time Adaptation with Refined Pseudo-labels and Latent Score Matching

Test-time adaptation (TTA) offers the potential to enhance model generalizability without relying on training data or retraining processes. However, TTA faces challenges under covariate shift, where discrepancies between the distributions of training and testing phases hinder model performance. This limitation stems from the fact that existing methods usually rely heavily on training data and fail to establish a good connection between the model and the marginal distribution of test data, resulting in reduced generalization ability. To mitigate this issue, we introduce a novel self-supervised framework that integrates latent score matching and pseudo-label refinement into the TTA paradigm to enhance the model's perception of the test data distribution. Our approach, Joint Test-time Adaptation with Refined Pseudo-labels and Latent Score Matching, reinterprets a classifier as a score estimator and trains it using pseudo-label refinement. This enables the model to better align with the test distribution through latent score matching, while simultaneously preserving discriminative performance via pseudo-label refinement. Extensive experiments across diverse architectures and benchmarks demonstrate that TAPS consistently outperforms state-of-the-art methods in terms of generalization performance under various distribution shifts.