Linearly Mapping from Image to Text Space

The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber NFRN50 Tuned : my motorcycle MAGMA (Ours) : my bike MAGMA (Rel.) : a 1973 Honda CB750 NFRN Random: the car on the road BEIT: the bike.' NFRN50: a motorcycle on a motorcycle. CLIP: my motorcycle. Ground Truth: A bicycle is standing next to a bed in a room. NFRN50 Tuned : my bike with a basket. MAGMA (Ours) : my bike MAGMA (Rel.) : a vintage bike vintage bike NFRN Random: a young boy in a blue shirt and a red hat on a bicycle BEIT: the bike. NFRN50: my bicycle with a basket on the front. CLIP: my bike, which is a fixed gear bike. Ground Truth: A man riding a wave on top of a white surfboard. NFRN50 Tuned : a surfer in the water MAGMA (Ours) : a surfer riding a wave. MAGMA (Rel.) : a surfer riding a wave NFRN Random: the view from the top of the mountain BEIT: a surfer riding a wave. NFRN50: a surfer in the water. CLIP: a surfer riding a wave. Ground Truth: A person standing on top of a beach flying a kite. NFRN50 Tuned : a man standing on the beach with a surfboard. MAGMA (Ours) : a kite on a beach. MAGMA (Rel.) : a man running on the beach NFRN Random: the new car in the garage BEIT: a man standing on a rock in the sea. NFRN50: a man surfing on the beach. CLIP: a man flying a kite on the beach. Ground Truth: A lady explains the process of milking a cow. NFRN50 Tuned : a horse and rider at the fair. MAGMA (Ours) : a woman on a stage with kids in the audience. MAGMA (Rel.) : a woman milking a cow in front of a group of children. NFRN Random: the new restaurant. BEIT: the crowd at the event. NFRN50: a crowd of people at a festival CLIP: the dairy cows. Ground Truth: A woman is taking notes in front of her laptop NFRN50 Tuned : the man who was shot and killed in the lobby of the building. MAGMA (Ours) : a student studying. MAGMA (Rel.) : a student working on a paper NFRN Random: the man who was killed in the attack. BEIT: the room where the meeting took place. NFRN50: the room where the meeting took place. CLIP: a student working on his laptop. Ground Truth: A couple of men standing on top of a field together. NFRN50 Tuned : the team before the match MAGMA (Ours) : men playing a game of sport. MAGMA (Rel.) : a men's lacrosse game at the University of New England NFRN Random: the new home of the football team BEIT: football player and football player during a training session. NFRN50: the team before the game. CLIP: the team during a practice session.