Lune

AAAI2024Top-tier venue

LINGO-Space: Language-Conditioned Incremental Grounding for Space

Dohyun Kim, Nayoung Oh, Deokmin Hwang, Daehyung Park

Abstract

We aim to solve the problem of spatially localizing composite instructions referring to space: space grounding. Compared to current instance grounding, space grounding is challenging due to the ill-posedness of identifying locations referred to by discrete expressions and the compositional ambiguity of referring expressions. Therefore, we propose a novel probabilistic space-grounding methodology (LINGO-Space) that accurately identifies a probabilistic distribution of space being referred to and incrementally updates it, given subsequent referring expressions leveraging configurable polar distributions. Our evaluations show that the estimation using polar distributions enables a robot to ground locations successfully through 20 table-top manipulation benchmark tests. We also show that updating the distribution helps the grounding method accurately narrow the referring space. We finally demonstrate the robustness of the space grounding with simulated manipulation and real quadruped robot navigation tasks. Code and videos are available at https://lingo-space.github.io .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Recipe for a General, Powerful, Scalable Graph Transformer cites Strategies for Pre-training Graph Neural Networks
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on5

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
  3. Strategies for Pre-training Graph Neural NetworksWeihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik et al.ICLR 2020 · 1,744 citations
  4. Recipe for a General, Powerful, Scalable Graph TransformerLadislav Rampásek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu et al.NeurIPS 2022 · 1,216 citations
  5. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language ModelsChan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao et al.ICCV 2023 · 685 citations

Related papers

  1. Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational DialogsBiswesh Mohapatra, Théo Charlot, Giovanni Duca, Mayank Palan et al.ACL Findings 2026
  2. Refer360∘^\circ: A Referring Expression Recognition Dataset in 360∘^\circ ImagesVolkan Cirik, Taylor Berg-Kirkpatrick, Louis-Philippe MorencyACL 2020 · 11 citations
  3. Behavioral Analysis of Vision-and-Language Navigation AgentsZijiao Yang, Arjun Majumdar, Stefan LeeCVPR 2023
  4. Grounding 3D Object Affordance with Language Instructions, Visual Observations and InteractionsHe Zhu, Quyu Kong, Kechun Xu, Xunlong Xia et al.CVPR 2025
  5. Naver: a Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic ReasoningZhixi Cai, Fucai Ke, Simindokht Jahangard, Maria Garcia de la Banda et al.ICCV 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines