Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction

Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot recognition capability, but still underperform in dense prediction tasks. Selfdistillation recently is emerging as a promising approach for fine-tuning VLMs to better adapt to local regions without requiring extensive annotations. However, previous stateof-the-art approaches often suffer from significant 'foreground bias', where models tend to wrongly identify background regions as foreground objects. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pretrained VLM representations. DenseVLM leverages the pretrained VLM to retrieve categories for unlabeled regions and then decouples the interference between foreground and background features. This separation ensures accurate region-category alignment while maintaining semantic distinctions during training. We show that DenseVLM can directly replace the original VLM in open-vocabulary object detection and image segmentation methods, leading to notable performance improvements. Furthermore, it exhibits promising zero-shot scalability when training on more extensive and diverse datasets. Our code is publicly available https://github.com/HVision-NKU/ DenseVLM .