Lune

ACL Findings2026

Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents

Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Shoubin Li, Qing Wang, Fanjiang Xu

Abstract

Embodied agents in safety-critical applications such as Vision-Language Navigation (VLN) rely on multiple interdependent capabilities (e.g., perception, memory, planning, decision), making failures difficult to localize and attribute. Existing testing methods are largely system-level and provide limited insight into which capability deficiencies cause task failures. We propose a capability-oriented testing approach that enables failure detection and attribution by combining (1) adaptive test case generation via seed selection and mutation, (2) capability oracles for identifying capabilityspecific errors, and (3) a feedback mechanism that attributes failures to capabilities and guides further test generation. Experiments show that our method discovers more failure cases and more accurately pinpoints capability-level deficiencies than state-of-the-art baselines, providing more interpretable and actionable guidance for improving embodied agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems cites Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Dreamwalker: Mental Planning for Continuous Vision-Language NavigationHanqing Wang, Wei Liang, Luc Van Gool, Wenguan WangICCV 2023 · 98 citations
  2. Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open SourceAnjiang Wei, Yinlin Deng, Chenyuan Yang, Lingming ZhangICSE 2022 · 91 citations
  3. Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective OptimizationFitash Ul Haq, Donghwan Shin, Lionel C. BriandICSE 2022 · 77 citations
  4. BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving SystemsMingfei Cheng, Yuan Zhou, Xiaofei XieISSTA 2023 · 59 citations
  5. Reinforced Structured State-Evolution for Vision-Language NavigationJinyu Chen, Chen Gao, Erli Meng, Qiong Zhang et al.CVPR 2022 · 37 citations

Related papers

  1. Generating Realistic and Diverse Tests for LiDAR-Based Perception SystemsGarrett Christian, Trey Woodlief, Sebastian G. ElbaumICSE 2023 · 11 citations
  2. MoDitector: Module-Directed Testing for Autonomous Driving SystemsRenzhi Wang, Mingfei Cheng, Xiaofei Xie, Yuan Zhou et al.ISSTA 2025 · 3 citations
  3. Think Twice, Act Once: Verifier-Guided Action Selection For Embodied AgentsNishad Singhi, Christian Bialas, Snehal Jauhri, Vignesh Prasad, Georgia Chalvatzaki, Marcus Rohrbach, Anna RohrbachCVPR Findings 2026
  4. RoboAgent: Chaining Basic Capabilities for Embodied Task PlanningPeiran Xu, Jiaqi Zheng, Yadong MuCVPR 2026 · 6 citations
  5. Fixed-Point Guided ADS Scenario Generation via Multi-modal LLM Reasoning and Software TestingXudong Zhang, Shihao Zhu, Yan CaiISSTA 2026
Dusk over the sea between two cliffs drawn in fine vertical lines