EMNLP Findings2020
ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
Hyounghun Kim, Abhaysinh Zala, Graham Burri, Hao Tan, Mohit Bansal
Abstract
For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as picking up objects and assembling them into a particular arrangement. We combine Vision-and-Language Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigationand-assembly task, named ARRAMON. During this task, the agent (similar to a PokéMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language (English) instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment. To support this task, we implement a 3D dynamic environment simulator and collect a dataset with human-written navigation and assembling instructions, and the corresponding ground truth trajectories. We also filter the collected instructions via a verification stage, leading to a total of 7.7K task instances (30.8K instructions and paths). We present results for several baseline models (integrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for future work. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TEACh: Task-Driven Embodied Agents That Chat cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue cites ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
- Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue cites TEACh: Task-Driven Embodied Agents That Chat
Cited by top-tier papers3
- TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange et al.AAAI 2022 · 251 citations
- A Meta-framework for Spatiotemporal Quantity Extraction from TextQiang Ning, Ben Zhou, Hao Wu, Haoruo Peng et al.ACL 2022 · 9 citations
- Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied DialogueAishwarya Padmakumar, Mert Inan, Spandana Gella, Patrick Lange et al.EMNLP 2023 · 1 citation
Builds on2
Related papers
- What You See Is What You Reach: Towards Spatial Navigation with High-Level Human InstructionsLingfeng Zhang, Haoxiang Fu, Xiaoshuai Hao, Shuyi Zhang et al.AAAI 2026 · 2 citations
- NaVLA: A Vision-Language-Audio-Action Model for Multimodal Instruction NavigationJugang Fan, Peihao Chen, Changhao Li, Qing Du et al.AAAI 2026
- NavA³: Understanding Any Instruction, Navigating Anywhere, Finding AnythingLingfeng Zhang, Xiaoshuai Hao, Yingbo Tang, Haoxiang Fu et al.ACL 2026 · 24 citations
- NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation PredictionFei Liu, Shichao Xie, Minghua Luo, Zedong Chu et al.CVPR 2026 · 16 citations
- Embodied Navigation Foundation ModelJiazhao Zhang, Anqi Li, Yunpeng Qi, Minghan Li et al.ICLR 2026 · 93 citations
