Lune

EMNLP Findings2024

Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation

Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, Tinne Tuytelaars

Abstract

This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finergrained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e., direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems. The project is now available at https://zehao- wang.github.io/navnuances.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation cites NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
  • Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation cites Learning Transferable Visual Models From Natural Language Supervision
  • How Much Can CLIP Benefit Vision-and-Language Tasks? cites Learning Transferable Visual Models From Natural Language Supervision
  • NavBench: Probing Multimodal Large Language Models for Embodied Navigation cites NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
  • Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents cites NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
  • NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. NavBench: Probing Multimodal Large Language Models for Embodied NavigationYanyuan Qiao, Haodong Hong, Wenqi Lyu, Dong An et al.NeurIPS 2025 · 27 citations
  2. Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language NavigationXiang Fang, Wanlong Fang, Changshuo WangNeurIPS 2025 · 24 citations
  3. CompassNav: Steering From Path Imitation to Decision Understanding In NavigationLinfeng Li, Jian Zhao, Yuan Xie, Xin Tan et al.ICLR 2026 · 14 citations
  4. Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation AgentsTianyi Ma, Yue Zhang, Zehao Wang, Parisa KordjamshidiACL 2026 · 3 citations

Builds on9

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. How Much Can CLIP Benefit Vision-and-Language Tasks?Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal et al.ICLR 2022 · 503 citations
  4. NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language ModelsGengze Zhou, Yicong Hong, Qi WuAAAI 2024 · 361 citations
  5. TEACh: Task-Driven Embodied Agents That ChatAishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange et al.AAAI 2022 · 251 citations

Related papers

  1. VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation AgentsXunyi Zhao, Gengze Zhou, Qi WuACL 2026 · 3 citations
  2. ULN: Towards Underspecified Vision-and-Language NavigationWeixi Feng, Tsu-Jui Fu, Yujie Lu, William Yang WangEMNLP 2022 · 2 citations
  3. GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMapFarzad Shami, Subhrasankha Dey, Nico Van de Weghe, Henrikki TenkanenACL 2026 · 1 citation
  4. Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor AreasRaphael Schumann, Stefan RiezlerACL 2022 · 38 citations
  5. Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' RuleShuhei Kurita, Kyunghyun ChoICLR 2021 · 29 citations
Dusk over the sea between two cliffs drawn in fine vertical lines