Distilling Out-of-Distribution Knowledge from Large Language Models for CLIP Generalization

Pretrained Vision-Language Models (VLMs) have demonstrated promising zero-shot capabilities; however, their performance on downstream tasks can be suboptimal due to the lack of domain-specific data during pretraining. To improve the performance of VLMs in specialized domains, parameter-efficient fine-tuning methods, such as Prompt Learning, have garnered significant attention. In this paper, we propose a plug-and-play method to enhance existing prompt learning approaches by synthesizing out-ofdistribution features from natural language prompts generated by Large Language Models (LLMs). Specifically, we first query an LLM to obtain textual knowledge for both seen and unseen samples. Then, we independently optimize a feature generator to synthesize pseudo-visual features for the base categories, using class-relative knowledge derived from the LLM-generated prompts. Subsequently, the well-trained generator can be used to generate synthesized visual features for novel classes based on their corresponding textual descriptions. Finally, we fine-tune the prompted-CLIP model by combining both real and pseudovisual features. Experiments on base-to-new generalization and cross-dataset transfer learning prove the effectiveness of our method.