Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as humanrobot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE 2 A 3 ) task, which aims to adjust the source-viewtrained model online during test time to anticipate targetview actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multiaction candidates and significant temporal-spatial interview gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-ofthe-art methods by a large margin. Code is available at https://github.com/ZhaofengSHI/DCPGN. Place Notebook Walk to Backpack Exocentric (Source-view) Observation Future Exo Video Egocentric (Target-view) Observation