Lune

ICCV2019Top-tier venue

PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings

Nicholas Rhinehart, Rowan McAllister, Kris Kitani, Sergey Levine

Abstract

For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information. Towards these capabilities, we present a probabilistic forecasting model of future interactions between a variable number of agents. We perform both standard forecasting and the novel task of conditional forecasting, which reasons about how all agents will likely respond to the goal of a controlled agent (here, the AV). We train models on real and simulated data to forecast vehicle trajectories given past positions and LI-DAR. Our evaluation shows that our model is substantially more accurate in multi-agent driving scenarios compared to existing state-of-the-art. Beyond its general ability to perform conditional forecasting queries, we show that our model's predictions of all agents improve when conditioned on knowledge of the AV's goal, further illustrating its capability to model agent interactions.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

62 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting cites Generative Hybrid Representations for Activity Forecasting With No-Regret Learning
  • Motion Transformer with Global Intention Localization and Local Movement Refinement cites Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset
  • Motion Transformer with Global Intention Localization and Local Movement Refinement cites DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
  • DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets cites Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset
  • HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction cites AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
  • HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction cites DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers62

  1. Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion DatasetScott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu et al.ICCV 2021 · 817 citations
  2. AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent ForecastingYe Yuan, Xinshuo Weng, Yanglan Ou, Kris KitaniICCV 2021 · 658 citations
  3. DenseTNT: End-to-end Trajectory Prediction from Dense Goal SetsJunru Gu, Chen Sun, Hang ZhaoICCV 2021 · 563 citations
  4. Motion Transformer with Global Intention Localization and Local Movement RefinementShaoshuai Shi, Li Jiang, Dengxin Dai, Bernt SchieleNeurIPS 2022 · 515 citations
  5. HiVT: Hierarchical Vector Transformer for Multi-Agent Motion PredictionZikang Zhou, Luyao Ye, Jianping Wang, Kui Wu et al.CVPR 2022 · 379 citations

Builds on3

  1. Deep Imitative Models for Flexible Inference, Planning, and ControlNicholas Rhinehart, Rowan McAllister, Sergey LevineICLR 2020 · 159 citations
  2. nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
  3. Generative Hybrid Representations for Activity Forecasting With No-Regret LearningJiaqi Guan, Ye Yuan, Kris M. Kitani, Nicholas RhinehartCVPR 2020

Related papers

  1. Scene Transformer: A unified architecture for predicting future trajectories of multiple agentsJiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang et al.ICLR 2022 · 194 citations
  2. Foresight in Motion: Reinforcing Trajectory Prediction with Reward HeuristicsMuleilan Pei, Shaoshuai Shi, Xuesong Chen, Xu Liu et al.ICCV 2025 · 7 citations
  3. MotionLM: Multi-Agent Motion Forecasting as Language ModelingAri Seff, Brian Cera, Dian Chen, Mason Ng et al.ICCV 2023 · 186 citations
  4. Prediction by Anticipation: An Action-Conditional Prediction Method based on Interaction LearningErshad Banijamali, Mohsen Rohani, Elmira Amirloo Abolfathi, Jun Luo et al.ICCV 2021 · 4 citations
  5. MUSE-VAE: Multi-Scale VAE for Environment-Aware Long Term Trajectory PredictionMihee Lee, Samuel S. Sohn, Seonghyeon Moon, Sejong Yoon et al.CVPR 2022 · 70 citations
Dusk over the sea between two cliffs drawn in fine vertical lines