EMNLP Findings2022
Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning
Yan Cao, Keting Lu, David DeFazio, Shiqi Zhang
Abstract
Vision-and-dialog navigation is a recent benchmark for evaluating the AI capabilities of perception, interaction, and decision making. While existing methods developed for this benchmark have demonstrated great successes, they mostly rely on large datasets, where data collection can be a challenge, and the learned policies are not adaptive to domain changes. In this paper, we focus on a new problem, referred to as goal-oriented vision-and-dialog navigation (GVDN), where an agent uses reinforcement learning techniques to compute dialog-navigation policies from trial and error. A robot conducts visual navigation to locate target objects, and can talk to a remote human operator as needed. Our remote human is able to provide guidance on navigation only if the robot correctly conveys its location through dialog. Experiments have been conducted using photo-realistic simulation environments. Results suggest that, our agent outperforms competitive baselines in success rate.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- MOSS: End-to-End Dialog System Framework with Modular SupervisionWeixin Liang, Youzhi Tian, Chengcai Chen, Zhou YuAAAI 2020 · 55 citations
- Guided Dialogue Policy Learning without Adversarial Learning in the LoopZiming Li, Sungjin Lee, Baolin Peng, Jinchao Li et al.EMNLP Findings 2020 · 18 citations
- Efficient Dialog Policy Learning by Reasoning with Contextual KnowledgeHaodi Zhang, Zhichao Zeng, Keting Lu, Kaishun Wu et al.AAAI 2022 · 14 citations
- Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-TrainingWeituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin et al.CVPR 2020
- Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future DirectionsJing Gu, Eliana Stefani, Qi Wu, Jesse Thomason et al.ACL 2022
Related papers
- DialNav: Multi-Turn Dialog Navigation with a Remote GuideLeekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi et al.ICCV 2025 · 1 citation
- Aerial Vision-and-Dialog NavigationYue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou et al.ACL Findings 2023 · 29 citations
- Self-Motivated Communication Agent for Real-World Vision-Dialog NavigationYi Zhu, Yue Weng, Fengda Zhu, Xiaodan Liang et al.ICCV 2021 · 41 citations
- Trajectory-Diversity-Driven Robust Vision-and-Language NavigationJiangyang Li, Cong Wan, SongLin Dong, Chenhao Ding, Qiang Wang, Zhiheng Ma, Yihong GongCVPR Findings 2026
- Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' RuleShuhei Kurita, Kyunghyun ChoICLR 2021 · 29 citations
