Lune

EMNLP Findings2020

Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions

Peter A. Jansen

Abstract

The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as "put a hot piece of bread on a plate". Currently, the best-performing models are able to complete less than 5% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to generate gold multi-step plans from language directives alone without any visual input in 26% of unseen cases. When a small amount of visual information is incorporated, namely the starting location in the virtual environment, our best-performing GPT-2 model successfully generates gold command sequences in 58% of cases. Our results suggest that contextualized language models may provide strong visual semantic planning modules for grounded virtual agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning
  • Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents cites The Curious Case of Neural Text Degeneration
  • PlanLLM: Video Procedure Planning with Refinable Large Language Models cites SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
  • PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning cites Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsWenlong Huang, Pieter Abbeel, Deepak Pathak, Igor MordatchICML 2022 · 1,539 citations
  2. PlanGenLLMs: A Modern Survey of LLM Planning CapabilitiesHui Wei, Zihao Zhang, Shenghua He, Tian Xia et al.ACL 2025 · 78 citations
  3. SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional VideosYulei Niu, Wenliang Guo, Long Chen, Xudong Lin et al.ICLR 2024 · 26 citations
  4. PlanLLM: Video Procedure Planning with Refinable Large Language ModelsDejie Yang, Zijing Zhao, Yang LiuAAAI 2025 · 8 citations
  5. PlaSma: Procedural Knowledge Models for Language-based Planning and Re-PlanningFaeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang et al.ICLR 2024 · 8 citations

Builds on2

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday TasksMohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk et al.CVPR 2020

Related papers

  1. LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied AgentsJae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim et al.ICLR 2024 · 49 citations
  2. R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action PlannerZiyi Bai, Hanxuan Li, Bin Fu, Chuyan Xiong et al.CVPR 2025
  3. Egocentric Planning for Scalable Embodied Task AchievementXiaotian Liu, Héctor Palacios, Christian MuiseNeurIPS 2023 · 9 citations
  4. GNN-Transformer Task Planning Enhanced with Semantic-Driven Data AugmentationSoojin Jeong, Seongwan Byeon, Sangwoo Kim, HyeokJun Kwon et al.AAAI 2025
  5. Factorizing Perception and Policy for Interactive Instruction FollowingKunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi et al.ICCV 2021 · 39 citations
Dusk over the sea between two cliffs drawn in fine vertical lines