EMNLP Findings2021
Compositional Data and Task Augmentation for Instruction Following
Abstract
Executing natural language instructions in a physically grounded domain requires a model that understands both spatial concepts such as left_of and above, and the compositional language used to identify landmarks and articulate instructions relative to them. In this paper, we study instruction understanding in the blocks world domain. Given an initial arrangement of blocks and a natural language instruction, the system executes the instruction by manipulating selected blocks. The highly compositional instructions are composed of atomic components and understanding these components is a necessary step to executing the instruction. We show that while end-to-end training (supervised only by the correct block location) fails to address the challenges of this task and performs poorly on instructions involving a single atomic component, knowledge-free auxiliary signals can be used to significantly improve performance by providing supervision for the instruction's components. Specifically, we generate signals that aim at helping the model gradually understand components of the compositional instructions, as well as those that help it better understand spatial concepts, and show their benefit to the overall task for two datasets and two state-of-the-art (SOTA) models, especially when the training data is limited-which is usual in such tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on1
Related papers
- Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks WorldGuande Wu, Chen Zhao, Cláudio T. Silva, He HeACL Findings 2024 · 3 citations
- Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction TasksYichen Jiang, Mohit BansalEMNLP 2021
- Generating Demonstrations for In-Context Compositional Generalization in Grounded Language LearningSam Spilsbury, Pekka Marttinen, Alexander IlinEMNLP 2024
- Mapping Natural Language Instructions to Mobile UI Action SequencesYang Li, Jiacong He, Xin Zhou, Yuan Zhang et al.ACL 2020 · 75 citations
- Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited DataAndrew C. Li, Toryn Q. Klassen, Andrew Wang, Parand A. Alamdari et al.NeurIPS 2025 · 5 citations
