Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

Large Language Models (LLMs) possess broad conceptual knowledge acquired through largescale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose LaViD-Language-to-Visual Knowledge Distillation-a simple and effective framework for transferring high-level semantic knowledge from a language-only teacher to a vision-only student model. Instead of relying on paired multimodal data, LaViD elicits conceptual signals from an LLM by prompting it to generate multiplechoice questions (MCQs) that probe semantic distinctions between visual classes. Each class is mapped to a soft label distribution over these MCQs, forming a rich conceptual signature that guides the student through an auxiliary distillation loss. Notably, despite using a languageonly teacher without access to image data, LaViD consistently outperforms recent methods like MaKD that distill from vision-language models across multiple fine-grained benchmarks. It also achieves competitive or superior performance compared to state-of-the-art visual distillation methods such as DKD and MLKD, with further gains when combined with logit standardization. On the Waterbirds dataset, LaViD substantially improves worst-group accuracy, demonstrating enhanced robustness to spurious correlations with distillation. Code is available at https: //github.com/lliangthomas/lavid . In this work, we introduce Language-to-Visual Knowledge Distillation (LaViD), a simple yet effective approach that distills general knowledge from text-only large language models (LLMs) into visual student models. Rather than relying on paired multimodal data or task-specific supervision, LaViD uses the broad world knowledge encoded in LLMs to provide conceptual guidance. It does so by eliciting structured and interpretable signals through multiple-choice questions that probe semantic distinctions between classes. This allows visual models to learn not just from labeled data,