Lune

AAAI2022Top-tier venue

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Dusan Grujicic, Thierry Deruyttere, Marie-Francine Moens, Matthew B. Blaschko

Abstract

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • nuScenes: A Multimodal Dataset for Autonomous Driving cites PointPainting: Sequential Fusion for 3D Object Detection
  • PointPainting: Sequential Fusion for 3D Object Detection cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving cites MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous DrivingYihong Tang, Haicheng Liao, Tong Nie, Junlin He et al.CVPR 2026 · 2 citations

Builds on8

  1. MDETR - Modulated Detection for End-to-End Multi-Modal UnderstandingAishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve et al.ICCV 2021 · 1,114 citations
  2. nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
  3. SE-SSD: Self-Ensembling Single-Stage Object Detector From Point CloudWu Zheng, Weiliang Tang, Li Jiang, Chi-Wing FuCVPR 2021
  4. PointPainting: Sequential Fusion for 3D Object DetectionSourabh Vora, Alex H. Lang, Bassam Helou, Oscar BeijbomCVPR 2020
  5. Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory PredictionAbduallah A. Mohamed, Kun Qian, Mohamed Elhoseiny, Christian G. ClaudelCVPR 2020

Related papers

  1. Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous VehiclesJiawei Liu, Xun Gong, Fen Fang, Muli Yang et al.CVPR 2026
  2. Where, What, Why: Towards Explainable Driver Attention PredictionYuchen Zhou, Jiayu Tang, Xiaoyan Xiao, Yueyao Lin et al.ICCV 2025 · 8 citations
  3. What Do People Want to Know about Artificial Intelligence (AI)? The Importance of Answering End-user Questions to Explain Autonomous Vehicle (AV) DecisionsSomayeh Molaei, Lionel Peter Robert, Nikola BanovicCSCW 2025 · 2 citations
  4. DriveVLN: Towards Mapless Vision-and-Language Navigation in Autonomous DrivingDongqian Guo, Haoran Wei, Wencheng Han, Runzhou Tao et al.CVPR 2026
  5. Think Before You Drive: World Model-Inspired Multimodal GroundingHaicheng Liao, Huanming Shen, Bonan Wang, yong kang li et al.CVPR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines