ACL Findings2022
VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator
Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gökhan Tür, Devi Parikh, Dilek Hakkani-Tur
Abstract
Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VIS-ITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical insights and improve performance on CVDN. VISITRON's ability to identify when to interact leads to a natural generalization of the gameplay mode introduced by Roman et al. (2020) for enabling the use of such models in different environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Unified Vision-Language Pre-Training for Image Captioning and VQA cites VideoBERT: A Joint Model for Video and Language Representation Learning
- Experience Grounds Language cites Unified Vision-Language Pre-Training for Image Captioning and VQA
- Experience Grounds Language cites VideoBERT: A Joint Model for Video and Language Representation Learning
- TEACh: Task-Driven Embodied Agents That Chat cites Experience Grounds Language
- TEACh: Task-Driven Embodied Agents That Chat cites Just Ask: An Interactive Learning Framework for Vision and Language Navigation
- Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation cites Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
- Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation cites History Aware Multimodal Transformer for Vision-and-Language Navigation
- History Aware Multimodal Transformer for Vision-and-Language Navigation cites Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
- Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation cites Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
- Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation cites History Aware Multimodal Transformer for Vision-and-Language Navigation
Cited by top-tier papers4
- History Aware Multimodal Transformer for Vision-and-Language NavigationShizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan LaptevNeurIPS 2021 · 427 citations
- TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange et al.AAAI 2022 · 251 citations
- Frequency-Enhanced Data Augmentation for Vision-and-Language NavigationKeji He, Chenyang Si, Zhihe Lu, Yan Huang et al.NeurIPS 2023 · 32 citations
- Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language NavigationYibo Cui, Liang Xie, Yakun Zhang, Meishan Zhang et al.ICCV 2023 · 31 citations
Builds on10
- VideoBERT: A Joint Model for Video and Language Representation LearningChen Sun, Austin Myers, Carl Vondrick, Kevin Murphy et al.ICCV 2019 · 1,396 citations
- Unified Vision-Language Pre-Training for Image Captioning and VQALuowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu et al.AAAI 2020 · 1,047 citations
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal GroundingAlexander Ku, Peter Anderson, Roma Patel, Eugene Ie et al.EMNLP 2020 · 208 citations
- Just Ask: An Interactive Learning Framework for Vision and Language NavigationTa-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim et al.AAAI 2020 · 88 citations
- Experience Grounds LanguageYonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas et al.EMNLP 2020 · 74 citations
Related papers
- Aerial Vision-and-Dialog NavigationYue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou et al.ACL Findings 2023 · 29 citations
- NDH-Full: Learning and Evaluating Navigational Agents on Full-Length DialogueHyounghun Kim, Jialu Li, Mohit BansalEMNLP 2021 · 11 citations
- SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language NavigationAbhinav Moudgil, Arjun Majumdar, Harsh Agrawal, Stefan Lee et al.NeurIPS 2021 · 88 citations
- Vision-Dialog Navigation by Exploring Cross-Modal MemoryYi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin et al.CVPR 2020
- Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' RuleShuhei Kurita, Kyunghyun ChoICLR 2021 · 29 citations
