Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs

Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to distinguish between visually similar subcategories-details that MLLMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in a fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boosts classification accuracy. We developed an automatic self-enhancing prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MLLMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consistently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP in average improves 13% over standard zero-shot classification and 3% over the best-performing baselines. Code is available at https://github.com/yq-hong/AutoSEP. Image MLLM Classification Prediction Zero-shot AutoSEP Image MLLM Description Generation Description MLLM Classification Prediction Optimized Description Generation Prompt Analyze the bird depicted in the image, focusing on the following details: * Beak: Describe its color, length, shape (e.g., curved, pointed, blunt), and any distinctive markings. * Head: Describe its color, any distinctive markings (e.g., eye stripe, eyebrow, throat patch), and the presence or absence of a crest. * Wings: Note the color, pattern (e.g., banded, barred, spotted), and any white patches or bands. * Tail: Describe its length, shape (e.g., graduated, rounded, square), and any distinctive patterns (e.g., banding, barring). * Body: Describe the overall body shape (e.g., slender, stout), the color of the chest and belly, and any distinctive markings. * Overall Size and Posture: Mention if the bird appears large or small, and describe its posture (e.g., upright, hunched, relaxed). Exclude any background details or other information from your description. Optimizing with unlabeled data