EMNLP Findings2020
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions
Abstract
The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as "put a hot piece of bread on a plate". Currently, the best-performing models are able to complete less than 5% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to generate gold multi-step plans from language directives alone without any visual input in 26% of unseen cases. When a small amount of visual information is incorporated, namely the starting location in the virtual environment, our best-performing GPT-2 model successfully generates gold command sequences in 58% of cases. Our results suggest that contextualized language models may provide strong visual semantic planning modules for grounded virtual agents.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents cites The Curious Case of Neural Text Degeneration
- PlanLLM: Video Procedure Planning with Refinable Large Language Models cites SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
- PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning cites Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Cited by top-tier papers6
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsWenlong Huang, Pieter Abbeel, Deepak Pathak, Igor MordatchICML 2022 · 1,539 citations
- PlanGenLLMs: A Modern Survey of LLM Planning CapabilitiesHui Wei, Zihao Zhang, Shenghua He, Tian Xia et al.ACL 2025 · 78 citations
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional VideosYulei Niu, Wenliang Guo, Long Chen, Xudong Lin et al.ICLR 2024 · 26 citations
- PlanLLM: Video Procedure Planning with Refinable Large Language ModelsDejie Yang, Zijing Zhao, Yang LiuAAAI 2025 · 8 citations
- PlaSma: Procedural Knowledge Models for Language-based Planning and Re-PlanningFaeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang et al.ICLR 2024 · 8 citations
Builds on2
Related papers
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied AgentsJae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim et al.ICLR 2024 · 49 citations
- R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action PlannerZiyi Bai, Hanxuan Li, Bin Fu, Chuyan Xiong et al.CVPR 2025
- Egocentric Planning for Scalable Embodied Task AchievementXiaotian Liu, Héctor Palacios, Christian MuiseNeurIPS 2023 · 9 citations
- GNN-Transformer Task Planning Enhanced with Semantic-Driven Data AugmentationSoojin Jeong, Seongwan Byeon, Sangwoo Kim, HyeokJun Kwon et al.AAAI 2025
- Factorizing Perception and Policy for Interactive Instruction FollowingKunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi et al.ICCV 2021 · 39 citations
