Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD are usually built on multi-modality representation learning, which pays heavily attention to inter-modality alignment rather than intra-modality alignment. In this paper, we propose a novel and effective multi-modal representation learning approach for GCD via Semi-Supervised Rate Reduction, called SSR 2 -GCD, to learn cross-modality representations with desired underlying structures via properly harnessing intra-modality alignment. Moreover, to boost knowledge transfer, we integrate the information from prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets, demonstrating superior performance of the proposed approach and verifying the importance of intramodality alignment. The code is available at: https: //github.com/hewei98/SSR2-GCD.