Lune

ACL Findings2021

Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

Yichi Zhang, Joyce Chai

摘要

Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the AL-FRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-toend architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT 1 (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 21 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • ALFWorld: Aligning Text and Embodied Environments for Interactive Learning 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • FILM: Following Instructions in Language with Modular Methods 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • FILM: Following Instructions in Language with Modular Methods 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Factorizing Perception and Policy for Interactive Instruction Following 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • Factorizing Perception and Policy for Interactive Instruction Following 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • PaLM-E: An Embodied Multimodal Language Model 引用了 Skill Induction and Planning with Latent Language
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 FILM: Following Instructions in Language with Modular Methods
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models 引用了 Skill Induction and Planning with Latent Language
  • TEACh: Task-Driven Embodied Agents That Chat 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Skill Induction and Planning with Latent Language 引用了 ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • Skill Induction and Planning with Latent Language 引用了 FILM: Following Instructions in Language with Modular Methods
  • Skill Induction and Planning with Latent Language 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions 引用了 ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper21

  1. PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch 等ICML 2023 · 2,601 次引用
  2. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language ModelsChan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao 等ICCV 2023 · 685 次引用
  3. TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange 等AAAI 2022 · 251 次引用
  4. FILM: Following Instructions in Language with Modular MethodsSo Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk 等ICLR 2022 · 189 次引用
  5. Skill Induction and Planning with Latent LanguagePratyusha Sharma, Antonio Torralba, Jacob AndreasACL 2022 · 127 次引用

它引用的 Paper5

  1. ALFWorld: Aligning Text and Embodied Environments for Interactive LearningMohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk 等ICLR 2021 · 819 次引用
  2. Situational Fusion of Visual Representation for Visual NavigationWilliam B. Shen, Danfei Xu, Yuke Zhu, Li Fei-Fei 等ICCV 2019 · 70 次引用
  3. Factorizing Perception and Policy for Interactive Instruction FollowingKunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi 等ICCV 2021 · 39 次引用
  4. Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level InstructionsPeter A. JansenEMNLP Findings 2020 · 33 次引用
  5. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday TasksMohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk 等CVPR 2020

相关 Paper

  1. Episodic Transformer for Vision-and-Language NavigationAlexander Pashevich, Cordelia Schmid, Chen SunICCV 2021 · 228 次引用
  2. Learning Grounded Action Abstractions from LanguageLionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel 等ICLR 2024 · 7 次引用
  3. ASHiTA: Automatic Scene-grounded HIerarchical Task AnalysisYun Chang, Leonor Fermoselle, Duy Ta, Bernadette Bucher 等CVPR 2025
  4. Possibility Before Utility: Learning And Using Hierarchical AffordancesRobby Costales, Shariq Iqbal, Fei ShaICLR 2022 · 5 次引用
  5. ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D ScenesRan Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng 等ICCV 2023 · 77 次引用
黄昏的海面,两侧是细线勾勒的悬崖