Modeling Collaborator: Enabling Subjective Vision Classification with Minimal Human Effort via LLM Tool-Use

From content moderation to wildlife conservation, the number of applications that require models to recognize nuanced or subjective visual concepts is growing. Traditionally, developing classifiers for such concepts requires substantial manual effort measured in hours, days, or even months to identify and annotate data needed for training. Even with recently proposed Agile Modeling techniques, which enable rapid bootstrapping of image classifiers, users are still required to spend 30 minutes or more of monotonous, repetitive data labeling just to train a single classifier. Drawing on Fiske's Cognitive Miser theory, we propose a new framework that alleviates manual effort by replacing human labeling with natural language interactions, reducing the total effort required to define a concept by an order of magnitude: from labeling 2,000 images to only 100 plus some natural language interactions. Our framework leverages recent advances in foundation models, both large language models and vision-language models, to carve out the concept space through conversation and by automatically labeling training data points. Most importantly, our framework eliminates the need for crowdsourced annotations. Moreover, our framework ultimately produces lightweight classification models that are deployable in cost-sensitive scenarios. Across 15 subjective concepts and across 2 public image classification datasets, our trained models outperform traditional Agile Modeling as well as state-of-the-art zero-shot classification models like ALIGN, CLIP, CuPL, and large visual question answering models like PaLI-X. * This work was done during an internship at Google. † This work was done during working at Google. Modeling Collaborator (Ours) Decision: Positive Reasons: The image shows a plate of food with chopsticks on it. The food is a salad with tuna, pineapple, avocado, and sesame seeds. The salad is on a plate with chopsticks on it. The dish is gourmet and contains tuna. PaLI-X (baseline) Prompt: Is this tuna gourmet? Justify your answer. PaLI: No, this tuna is not gourmet. It is a simple salad with tuna, sesame seeds, and vegetables. It is not a fancy or elaborate dish. Prompt: Could this be an image of gourmet tuna? PaLI: Yes, this could be an image of gourmet tuna. Prompt: Is this tuna gourmet? Are you sure? PaLI: I'm not sure if this tuna is gourmet. It looks like it could be, but it's hard to say for sure without tasting it.