Lune

CVPR Findings2026

VR-CLIP: Visual Refinement of CLIP for Zero-Shot Semantic Segmentation

Haitao Jiang, Xu Li, Yuanyang Cao, Ying Zhang, Jianji Wang

Abstract

Zero-shot semantic segmentation has gained significant attention with the emergence of vision-language models such as CLIP. Most existing methods primarily focus on improving generalization or designing more efficient decoders. However, they remain constrained by CLIP's image-level visual features, which limit pixel-level discriminability and make dense predictions heavily reliant on the decoder. To address these issues, we are the first to explore optimizing visual features for zero-shot semantic segmentation task. Building on this idea, we propose VR-CLIP (Visual Refinement of CLIP), a one-stage framework that refines CLIP's visual representations within the encoder from both semantic and spatial perspectives. Specifically, the Neighborhood Spatial Awareness (NSA) module enhances visual continuity through local spatial modeling, while the Semantic-Guided Visual Refinement (SGR) integrates class-level semantic cues to strengthen feature discriminability and preserve CLIP's inherent visual-linguistic alignment. Moreover, a parameter-sharing design within SGR further promotes generalization to unseen categories. Extensive experiments on PASCAL VOC 2012, COCO-Stuff 164K, and PASCAL-Context demonstrate that VR-CLIP consistently outperforms state-of-the-art methods under both inductive and transductive settings.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation cites SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
  • SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on20

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. SegFormer: Simple and Efficient Design for Semantic Segmentation with TransformersEnze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar et al.NeurIPS 2021 · 9,661 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. SegNeXt: Rethinking Convolutional Attention Design for Semantic SegmentationMeng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu et al.NeurIPS 2022 · 1,385 citations

Related papers

  1. Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic SegmentationYunheng Li, Zhong-Yu Li, Quan-Sheng Zeng, Qibin Hou et al.ICML 2024 · 27 citations
  2. Exploring Regional Clues in CLIP for Zero-Shot Semantic SegmentationYi Zhang, Meng-Hao Guo, Miao Wang, Shi-Min HuCVPR 2024 · 20 citations
  3. Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation OnlyJun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem et al.ICCV 2023 · 60 citations
  4. Object-Centric Refinement for Enhanced Zero-Shot SegmentationSrinivasa Rao Nandam, Sara Atito Ali, Zhenhua Feng, Josef Kittler et al.ICLR 2026 · 5 citations
  5. Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic SegmentationWenhao Xu, Rongtao Xu, Changwei Wang, Shibiao Xu et al.AAAI 2024 · 21 citations
Dusk over the sea between two cliffs drawn in fine vertical lines