Instruction Decomposition and Action Alignment for Vision-Language Navigation
Zihao Xin, Wentong Li, Yixuan Jiang, Bin Wang, Piji Li, Jianke Zhu, Jie Qin, Sheng-Jun Huang
Abstract
Vision-and-Language Navigation (VLN) empowered by Multimodal Large Language Models (MLLMs) is promise, yet remains challenged by long-horizon tasks with complex user instructions. Existing approaches that continuously condition on full instructions incur high latency due to abundant visual tokens and exacerbates instruction interference, where irrelevant text noise induces hallucinations. To address these limitations, we propose IDEAL-VLN ( Instruction DEcomposition and Action aLignment ), a novel paradigm that reformulates navigation as a causal inference chain. We decompose the task into two sequential steps: Semantic Anchoring and Action Alignment. We adopt a Think-Before-Act mechanism that first infers the immediate semantic anchor from the global context and then generates actions conditioned solely on this anchor. This design constructs an explicit information bottleneck, suppressing spurious correlations from irrelevant instruction. Moreover, to alleviate cognitive collapse and limited exploration during training, we introduce a hierarchical correction framework that combines semantic-level thought correction with a spatially-aware adaptive intervention strategy. This strategy adjusts expert intervention probability based on geodesic distance, effectively defining a semantic safety boundary. To support this paradigm, we contribute the Instruction-Aligned Navigation Dataset containing 160K image-text pairs. Extensive experiments demonstrate that IDEAL-VLN achieves state-of-the-art performance and robustness across major benchmarks while significantly reducing inference costs.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 48b9fbaf-cd86-4a2b-98de-80ecff1d5ff5Builds on24
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMsPeter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo et al.NeurIPS 2024 · 1,004 citations
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal GroundingAlexander Ku, Peter Anderson, Roma Patel, Eugene Ie et al.EMNLP 2020 · 208 citations
Related papers
- SeqWalker: Sequential-Horizon Vision-and-Language Navigation with Hierarchical PlanningZebin Han, Xudong Wang, Baichen Liu, Qi Lyu et al.AAAI 2026 · 2 citations
- Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language NavigationShuo Wang, Yongcai Wang, Wanting Li, Xudong Cai et al.NeurIPS 2025 · 28 citations
- JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language NavigationShuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong et al.ICLR 2026 · 124 citations
- ADAPT: Vision-Language Navigation with Modality-Aligned Action PromptsBingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang et al.CVPR 2022 · 45 citations
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language NavigationYu Zhong, Zihao Zhang, Rui Zhang, Lingdong Huang et al.AAAI 2026
