Test-time Adaptation for Cross-modal Retrieval with Query Shift

The success of most existing cross-modal retrieval methods heavily relies on the assumption that the given queries follow the same distribution of the source domain. However, such an assumption is easily violated in real-world scenarios due to the complexity and diversity of queries, thus leading to the query shift problem. Specifically, query shift refers to the online query stream originating from the domain that follows a different distribution with the source one. In this paper, we observe that query shift would not only diminish the uniformity (namely, withinmodality scatter) of the query modality but also amplify the gap between query and gallery modalities. Based on the observations, we propose a novel method dubbed Test-time adaptation for Cross-modal Retrieval (TCR). In brief, TCR employs a novel module to refine the query predictions (namely, retrieval results of the query) and a joint objective to prevent query shift from disturbing the common space, thus achieving online adaptation for the cross-modal retrieval models with query shift. Expensive experiments demonstrate the effectiveness of the proposed TCR against query shift. Code is available at https://github.com/XLearning-SCU/2025-ICLR-TCR . * Corresponding author. (a)Dominant Paradigm (b)Query Shift (c)Observations Pre-trained Models Fine-tune Zero-shot Rib knit wool beanie in 'anthracite' grey. Rubberized logo in red and white at side. Query from Personalized Domains Query from Scarce Domains Long sleeve cotton twill blazer in off-white. Raw edges throughout. Notched lapel collar.