Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models

Building state-of-the-art Vision-Language Models (VLMs) with strong captioning capabilities typically necessitates training on billions of high-quality image-text pairs, requiring millions of GPU hours. This paper introduces the Vision-Language-Vision (VLV) auto-encoder framework, which strategically leverages key pretrained components: a vision encoder, the decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large Language Model (LLM). Specifically, we establish an information bottleneck by regularizing the language representation space, achieved through freezing the pretrained T2I diffusion decoder. Our VLV pipeline effectively distills knowledge from the text-conditioned diffusion model using continuous embeddings, demonstrating comprehensive semantic understanding via high-quality reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the intermediate language representations into detailed descriptions, we construct a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and significantly reduces data requirements; by primarily utilizing single-modal images for training and maximizing the utility of existing pretrained models (image encoder, T2I diffusion model, and LLM), it circumvents the need for massive paired image-text datasets, keeping the total training expenditure under $1,000 USD.

A wide-eyed orange cat and a scruffy white dog sit stiffly on a shredded green couch in a messy, cartoon-style living room. The couch cushions are ripped open, spilling stuffing onto the floor and into the air. Both animals look stunned, as if caught in the act of causing the destruction. Behind them, a bookshelf packed with colorful books leans slightly, and a crooked picture frame hangs on a cracked green wall. Debris floats midair, suggesting a recent burst of chaotic energy. The wooden floor is littered with torn paper, chewed-up fabric, and scattered household items. The lighting is warm and even, highlighting the exaggerated cartoon features and the humorous tension in the scene.

An eye-level, full shot shows a green couch with an orange cat and a white dog sitting on it . The couch is covered with a green throw blanket. The orange cat is sitting on the left side of the couch, looking to the left. The white dog is sitting on the right side of the couch, looking to the right. The dog has a red collar around its neck. The couch is in a room with green walls. There is a wooden bookshelf on the left side of the couch. The bookshelf is filled with books and other items. There is a lamp on the bookshelf. There is a framed picture on the wall above the couch. There is a red rug on the floor in front of the couch. There is a wooden table in front of the couch. The table is covered with papers and other items. The lighting in the room is bright. The mood of the image is cheerful. The colors in the image are bright and cheerful. The textures in the image are smooth and soft. The shadows in the image are soft and subtle. The overall atmosphere of the image is warm and inviting. The image is well-composed and visually appealing. The details in the image are clear and well-defined.