Learning to Memorize with Attributive and Associative Memory for Online Test-Time Adaptation of Vision-Language Models

Memory-based test-time adaptation (TTA) assigns streaming test samples into class-specific memory slots based on pseudo-labels predicted by models like CLIP, and retrieves them to facilitate subsequent predictions under distribution shift. However, this process introduces two challenges: (1) Each sample is hard-assigned to a single class based on CLIP's prediction, where inaccurate CLIP prediction leads to memory contamination that biases subsequent prediction. (2) Samples are evicted under biased selection due to fixed memory capacity, which risks discarding informative samples and undermining the efficacy of the memory. To address these challenges, we propose A²Memory (Attributive-Associative Memory for Test-time Adaptation). For challenge (1), we propose Attribute-centric Memory Construction that builds prior textual representations from class-shared representative and diverse visual attributes, and applies soft assignment to generate surrogate visual representations. For challenge (2), we design Class-wise Associative Memory that dynamically compresses streaming samples into fixed-capacity memory through gradient-based optimization and data-dependent retention, then retrieves sample-adaptive class prototypes for reliable inference. Extensive experiments demonstrate consistent improvements over state-of-the-art methods across 15 benchmarks.