Lune

EMNLP Findings2021

Does Vision-and-Language Pretraining Improve Lexical Grounding?

Tian Yun, Chen Sun, Ellie Pavlick

Abstract

Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions cites Language Models are Few-Shot Learners
  • If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning to Imagine: Visually-Augmented Natural Language Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Can Language Models Understand Physical Concepts? cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Can Language Models Understand Physical Concepts? cites Language Models are Few-Shot Learners
  • Can Language Models Understand Physical Concepts? cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Can Language Models Understand Physical Concepts?Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng et al.EMNLP 2023 · 13 citations
  2. Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter ItYulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli et al.NeurIPS 2025 · 11 citations
  3. Learning to Imagine: Visually-Augmented Natural Language GenerationTianyi Tang, Yushuo Chen, Yifan Du, Junyi Li et al.ACL 2023 · 7 citations
  4. If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept DescriptionsReza Esfandiarpoor, Cristina Menghini, Stephen H. BachEMNLP 2024 · 3 citations

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  4. VideoBERT: A Joint Model for Video and Language Representation LearningChen Sun, Austin Myers, Carl Vondrick, Kevin Murphy et al.ICCV 2019 · 1,396 citations
  5. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations

Related papers

  1. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-TrainingGen Li, Nan Duan, Yuejian Fang, Ming Gong et al.AAAI 2020 · 966 citations
  2. Understanding ME? Multimodal Evaluation for Fine-grained Visual CommonsenseZhecan Wang, Haoxuan You, Yicheng He, Wenhao Li et al.EMNLP 2022 · 2 citations
  3. Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCRZhenyang Li, Yangyang Guo, Kejie Wang, Xiaolin Chen et al.ACM MM 2023 · 11 citations
  4. Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language ModelsAryan Roy, Zekun Wang, Christopher J. MacLellanACL Findings 2026
  5. VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
Dusk over the sea between two cliffs drawn in fine vertical lines