ACL Findings2024
Exploring Spatial Schema Intuitions in Large Language and Vision Models
Abstract
Despite the ubiquity of large language models (LLMs) in AI research, the question of embodiment in LLMs remains underexplored, distinguishing them from embodied systems in robotics where sensory perception directly informs physical action.Our investigation navigates the intriguing terrain of whether LLMs, despite their non-embodied nature, effectively capture implicit human intuitions about fundamental, spatial building blocks of language.We employ insights from spatial cognitive foundations developed through early sensorimotor experiences, guiding our exploration through the reproduction of three psycholinguistic experiments.Surprisingly, correlations between model outputs and human responses emerge, revealing adaptability without a tangible connection to embodied experiences.Notable distinctions include polarized language model responses and reduced correlations in vision language models.This research contributes to a nuanced understanding of the interplay between language, spatial experiences, and the computations made by large language models. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on4
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject StudiesGati V. Aher, Rosa I. Arriaga, Adam Tauman KalaiICML 2023 · 651 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- What's "up" with vision-language models? Investigating their struggle with spatial reasoningAmita Kamath, Jack Hessel, Kai-Wei ChangEMNLP 2023 · 31 citations
Related papers
- Can LLMs Learn to Map the World from Local Descriptions?Sirui Xia, Aili Chen, Xintao Wang, Tinghui Zhu et al.ACL 2026 · 2 citations
- With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language ModelsTyler Loakman, Yucheng Li, Chenghua LinEMNLP 2024 · 1 citation
- Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?Zhihui Yang, Yupei Wang, Kaijie Mo, Zhe Zhao et al.EMNLP Findings 2025 · 2 citations
- Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from DemonstrativesYu Wang, Emmanuele Chersoni, Chu-Ren HuangACL 2026
- Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from PsychometricsWenrui Xu, Dalin Lyu, Weihang Wang, Jie Feng et al.ACL 2025 · 19 citations
