ViTPrompt: Training-Free Prompt Refinement with Visual Tokens for Open-Vocabulary Detection

Test-Time Adaptive Object Detection (TTAOD) aims to maintain detection performance under distribution shifts without retraining. While recent vision-language models enable open-vocabulary detection, existing TTAOD methods-whether closed-set or open-vocabulary-focus solely on improving classification confidence and ignore the degradation of bounding box localization. To address this critical gap, we propose ViTPrompt (Visual Token-Prompting), a training-free framework that jointly refines both bounding boxes and class scores at test time.

Our key insight is to augment the original text prompt with instance-aware visual tokens extracted from highconfidence detections in an initial forward pass. This enhanced prompt is then fed into a second inference stage, where the cross-modal decoder produces updated box coordinates and more accurate classification logits through enriched semantic context. ViTPrompt requires no backpropagation, parameter updates, or external memory, making it highly efficient for real-time deployment. Experiments on multiple out-of-distribution benchmarks show that ViT-Prompt achieves state-of-the-art performance, with consistent gains in both localization accuracy and classification fidelity , demonstrating its effectiveness as a holistic solution for open-vocabulary TTAOD. We will release our code at https://github.com/buerzlh/Test-time-Adaptation-for-Object-Detection.