Lune

ICCV2021Top-tier venue

FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, Alex Kendall

Abstract

Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird’s-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird’s-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inherent stochastic nature of the future solely from camera driving data in an end-to-end manner, without relying on HD maps, and predicts multimodal future trajectories. We show that our model outperforms previous prediction baselines on the NuScenes and Lyft datasets. The code and trained models are available at https://github.com/wayveai/fiery.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

93 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • nuScenes: A Multimodal Dataset for Autonomous Driving cites CoverNet: Multimodal Behavior Prediction Using Trajectory Sets
  • CoverNet: Multimodal Behavior Prediction Using Trajectory Sets cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation cites PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings
  • MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • CoverNet: Multimodal Behavior Prediction Using Trajectory Sets cites PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings
  • VAD: Vectorized Scene Representation for Efficient Autonomous Driving cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • VAD: Vectorized Scene Representation for Efficient Autonomous Driving cites VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation
  • VAD: Vectorized Scene Representation for Efficient Autonomous Driving cites CoverNet: Multimodal Behavior Prediction Using Trajectory Sets
  • PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability cites nuScenes: A Multimodal Dataset for Autonomous Driving
  • Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability cites VAD: Vectorized Scene Representation for Efficient Autonomous Driving
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers93

  1. VAD: Vectorized Scene Representation for Efficient Autonomous DrivingBo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao et al.ICCV 2023 · 602 citations
  2. PETRv2: A Unified Framework for 3D Perception from Multi-Camera ImagesYingfei Liu, Junjie Yan, Fan Jia, Shuailin Li et al.ICCV 2023 · 513 citations
  3. Vista: A Generalizable Driving World Model with High Fidelity and Versatile ControllabilityShenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta et al.NeurIPS 2024 · 403 citations
  4. SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous DrivingYi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu et al.ICCV 2023 · 380 citations
  5. OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy PredictionYunpeng Zhang, Zheng Zhu, Dalong DuICCV 2023 · 354 citations

Builds on7

  1. PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent SettingsNicholas Rhinehart, Rowan McAllister, Kris Kitani, Sergey LevineICCV 2019 · 407 citations
  2. nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
  3. VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized RepresentationJiyang Gao, Chen Sun, Hang Zhao, Yi Shen et al.CVPR 2020
  4. MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View MapsPengxiang Wu, Siheng Chen, Dimitris N. MetaxasCVPR 2020
  5. CoverNet: Multimodal Behavior Prediction Using Trajectory SetsTung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom et al.CVPR 2020

Related papers

  1. Dualad: Disentangling the Dynamic and Static World for End-to-End DrivingSimon Doll, Niklas Hanselmann, Lukas Schneider, Richard Schulz et al.CVPR 2024
  2. LaPred: Lane-Aware Prediction of Multi-Modal Future Trajectories of Dynamic AgentsByeoungdo Kim, SeongHyeon Park, Seokhwan Lee, Elbek Khoshimjonov et al.CVPR 2021
  3. Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View With a Reachability PriorOsama Makansi, Özgün Çiçek, Kevin Buchicchio, Thomas BroxCVPR 2020
  4. Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving ApplicationsJunyi Ma, Xieyuanli Chen, Jiawei Huang, Jingyi Xu et al.CVPR 2024 · 28 citations
  5. DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous DrivingChen Shi, Shaoshuai Shi, Kehua Sheng, Bo Zhang et al.ICCV 2025
Dusk over the sea between two cliffs drawn in fine vertical lines