Lune

ACL Findings2021

Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

Yichi Zhang, Joyce Chai

Abstract

Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the AL-FRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-toend architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT 1 (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

21 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ALFWorld: Aligning Text and Embodied Environments for Interactive Learning cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • FILM: Following Instructions in Language with Modular Methods cites ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • FILM: Following Instructions in Language with Modular Methods cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Factorizing Perception and Policy for Interactive Instruction Following cites ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • Factorizing Perception and Policy for Interactive Instruction Following cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • PaLM-E: An Embodied Multimodal Language Model cites Skill Induction and Planning with Latent Language
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models cites FILM: Following Instructions in Language with Modular Methods
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models cites Skill Induction and Planning with Latent Language
  • TEACh: Task-Driven Embodied Agents That Chat cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Skill Induction and Planning with Latent Language cites ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
  • Skill Induction and Planning with Latent Language cites FILM: Following Instructions in Language with Modular Methods
  • Skill Induction and Planning with Latent Language cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
  • Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers21

  1. PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
  2. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language ModelsChan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao et al.ICCV 2023 · 685 citations
  3. TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange et al.AAAI 2022 · 251 citations
  4. FILM: Following Instructions in Language with Modular MethodsSo Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk et al.ICLR 2022 · 189 citations
  5. Skill Induction and Planning with Latent LanguagePratyusha Sharma, Antonio Torralba, Jacob AndreasACL 2022 · 127 citations

Builds on5

  1. ALFWorld: Aligning Text and Embodied Environments for Interactive LearningMohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk et al.ICLR 2021 · 819 citations
  2. Situational Fusion of Visual Representation for Visual NavigationWilliam B. Shen, Danfei Xu, Yuke Zhu, Li Fei-Fei et al.ICCV 2019 · 70 citations
  3. Factorizing Perception and Policy for Interactive Instruction FollowingKunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi et al.ICCV 2021 · 39 citations
  4. Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level InstructionsPeter A. JansenEMNLP Findings 2020 · 33 citations
  5. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday TasksMohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk et al.CVPR 2020

Related papers

  1. Episodic Transformer for Vision-and-Language NavigationAlexander Pashevich, Cordelia Schmid, Chen SunICCV 2021 · 228 citations
  2. Learning Grounded Action Abstractions from LanguageLionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel et al.ICLR 2024 · 7 citations
  3. ASHiTA: Automatic Scene-grounded HIerarchical Task AnalysisYun Chang, Leonor Fermoselle, Duy Ta, Bernadette Bucher et al.CVPR 2025
  4. Possibility Before Utility: Learning And Using Hierarchical AffordancesRobby Costales, Shariq Iqbal, Fei ShaICLR 2022 · 5 citations
  5. ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D ScenesRan Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng et al.ICCV 2023 · 77 citations
Dusk over the sea between two cliffs drawn in fine vertical lines