CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation

Existing semantic segmentation approaches are often limited by costly pixel-wise annotations and predefined classes. In this work, we present CLIP-S 4 that leverages self-supervised pixel representation learning and visionlanguage models to enable various semantic segmentation tasks (e.g., unsupervised, transfer learning, languagedriven segmentation) without any human annotations and unknown class information. We first learn pixel embeddings with pixel-segment contrastive learning from different augmented views of images. To further improve the pixel embeddings and enable language-driven semantic segmentation, we design two types of consistency guided by visionlanguage models: 1) embedding consistency, aligning our pixel embeddings to the joint feature space of a pre-trained vision-language model, CLIP [34]; and 2) semantic consistency, forcing our model to make the same predictions as CLIP over a set of carefully designed target classes with both known and unknown prototypes. Thus, CLIP-S 4 enables a new task of class-free semantic segmentation where no unknown class information is needed during training. As a result, our approach shows consistent and substantial performance improvement over four popular benchmarks compared with the state-of-the-art unsupervised and languagedriven semantic segmentation methods. More importantly, our method outperforms these methods on unknown class recognition by a large margin. Image MaskCLIP MaskCLIP+ CLIP-S⁴ (a) Pixel embeddings generated by different CLIP-based unsupervised approaches (c) CLIP-S⁴ aligns the pixel embeddings and their semantics with CLIP feature space CLIP (Text) CLIP (Pixel) CLIP-S⁴ (Pixel w/o Alignment) Align ci cj ci cj CLIP (Text) CLIP (Pixel) CLIP-S⁴ (Pixel w/ Alignment) ci unknown Known: airplane Unknown: moon (b) Semantic segmentation generated by different CLIP-based unsupervised approaches MaskCLIP MaskCLIP+ CLIP-S⁴