Predicting Physical World Destinations for Commands Given to Self-Driving Cars
Dusan Grujicic, Thierry Deruyttere, Marie-Francine Moens, Matthew B. Blaschko
Abstract
In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- nuScenes: A Multimodal Dataset for Autonomous Driving cites PointPainting: Sequential Fusion for 3D Object Detection
- PointPainting: Sequential Fusion for 3D Object Detection cites nuScenes: A Multimodal Dataset for Autonomous Driving
- E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving cites MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Cited by top-tier papers1
Builds on8
- MDETR - Modulated Detection for End-to-End Multi-Modal UnderstandingAishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve et al.ICCV 2021 · 1,114 citations
- nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
- SE-SSD: Self-Ensembling Single-Stage Object Detector From Point CloudWu Zheng, Weiliang Tang, Li Jiang, Chi-Wing FuCVPR 2021
- PointPainting: Sequential Fusion for 3D Object DetectionSourabh Vora, Alex H. Lang, Bassam Helou, Oscar BeijbomCVPR 2020
- Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory PredictionAbduallah A. Mohamed, Kun Qian, Mohamed Elhoseiny, Christian G. ClaudelCVPR 2020
Related papers
- Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous VehiclesJiawei Liu, Xun Gong, Fen Fang, Muli Yang et al.CVPR 2026
- Where, What, Why: Towards Explainable Driver Attention PredictionYuchen Zhou, Jiayu Tang, Xiaoyan Xiao, Yueyao Lin et al.ICCV 2025 · 8 citations
- What Do People Want to Know about Artificial Intelligence (AI)? The Importance of Answering End-user Questions to Explain Autonomous Vehicle (AV) DecisionsSomayeh Molaei, Lionel Peter Robert, Nikola BanovicCSCW 2025 · 2 citations
- DriveVLN: Towards Mapless Vision-and-Language Navigation in Autonomous DrivingDongqian Guo, Haoran Wei, Wencheng Han, Runzhou Tao et al.CVPR 2026
- Think Before You Drive: World Model-Inspired Multimodal GroundingHaicheng Liao, Huanming Shen, Bonan Wang, yong kang li et al.CVPR 2026
