ACL Findings2021
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
Yichi Zhang, Joyce Chai
摘要
Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the AL-FRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-toend architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT 1 (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 21 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- FILM: Following Instructions in Language with Modular Methods 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- FILM: Following Instructions in Language with Modular Methods 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Factorizing Perception and Policy for Interactive Instruction Following 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Factorizing Perception and Policy for Interactive Instruction Following 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- PaLM-E: An Embodied Multimodal Language Model 引用了 Skill Induction and Planning with Latent Language
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 FILM: Following Instructions in Language with Modular Methods
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 Skill Induction and Planning with Latent Language
- TEACh: Task-Driven Embodied Agents That Chat 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Skill Induction and Planning with Latent Language 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Skill Induction and Planning with Latent Language 引用了 FILM: Following Instructions in Language with Modular Methods
- Skill Induction and Planning with Latent Language 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
引用它的顶会 Paper21
- PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch 等ICML 2023 · 2,601 次引用
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language ModelsChan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao 等ICCV 2023 · 685 次引用
- TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange 等AAAI 2022 · 251 次引用
- FILM: Following Instructions in Language with Modular MethodsSo Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk 等ICLR 2022 · 189 次引用
- Skill Induction and Planning with Latent LanguagePratyusha Sharma, Antonio Torralba, Jacob AndreasACL 2022 · 127 次引用
它引用的 Paper5
- ALFWorld: Aligning Text and Embodied Environments for Interactive LearningMohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk 等ICLR 2021 · 819 次引用
- Situational Fusion of Visual Representation for Visual NavigationWilliam B. Shen, Danfei Xu, Yuke Zhu, Li Fei-Fei 等ICCV 2019 · 70 次引用
- Factorizing Perception and Policy for Interactive Instruction FollowingKunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi 等ICCV 2021 · 39 次引用
- Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level InstructionsPeter A. JansenEMNLP Findings 2020 · 33 次引用
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday TasksMohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk 等CVPR 2020
相关 Paper
- Episodic Transformer for Vision-and-Language NavigationAlexander Pashevich, Cordelia Schmid, Chen SunICCV 2021 · 228 次引用
- Learning Grounded Action Abstractions from LanguageLionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel 等ICLR 2024 · 7 次引用
- ASHiTA: Automatic Scene-grounded HIerarchical Task AnalysisYun Chang, Leonor Fermoselle, Duy Ta, Bernadette Bucher 等CVPR 2025
- Possibility Before Utility: Learning And Using Hierarchical AffordancesRobby Costales, Shariq Iqbal, Fei ShaICLR 2022 · 5 次引用
- ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D ScenesRan Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng 等ICCV 2023 · 77 次引用
