SaTeen: Learning Structural Alignment for Continual Test-Time Adaptation

Test-Time Adaptation (TTA) aims to reconcile model generalization in the presence of distribution shifts. Current TTA methods usually leverage sample uncertainty to select reliable samples for model adjustment via entropy minimization (EM). However, sample uncertainty often relies on a plausible metric and leaves many unreliable samples into EM process, potentially leading to model collapse. Importantly, these excluded samples incur biased data features of the shifted distribution in TTA. This paper introduces SaTeen, a S tructural A lignment-based Te st-Tim e Adaptatio n method that performs two-fold aligning the structures of test samples with the reliable reference structures. Specifically, the two-fold alignments are 1) Intra-sample structure alignment, where SaTeen maximizes cross-entropy discrepancy between a sample (reference) and its structure-disrupted counterpart, with the assumption of stable dominant features; 2) Inter-sample structure alignment, where SaTeen minimizes the reconstruction error of test samples in the reference subspace spanned by the Incremental PCA on reliable samples, with the assumption of stale intrinsic data manifold. Our extensive experiments demonstrate that SaTeen achieves the state-of-the-art performance across various scenarios for both TTA and continual TTA.