Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, Christopher Pal
Abstract
Robust multi-agent trajectory prediction is essential for the safe control of robotic systems. A major challenge is to efficiently learn a representation that approximates the true joint distribution of contextual, social, and temporal information to enable planning. We propose Latent Variable Sequential Set Transformers which are encoder-decoder architectures that generate scene-consistent multi-agent trajectories. We refer to these architectures as "AutoBots". The encoder is a stack of interleaved temporal and social multi-head self-attention (MHSA) modules which alternately perform equivariant processing across the temporal and social dimensions. The decoder employs learnable seed parameters in combination with temporal and social MHSA modules allowing it to perform inference over the entire future scene in a single forward pass efficiently. AutoBots can produce either the trajectory of one ego-agent or a distribution over the future trajectories for all agents in the scene. For the single-agent prediction case, our model achieves top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results on the Argoverse vehicle prediction challenge. In the multi-agent setting, we evaluate on the synthetic partition of TrajNet++ dataset to showcase the model's socially-consistent predictions. We also demonstrate our model on general sequences of sets and provide illustrative experiments modelling the sequential structure of the multiple strokes that make up symbols in the Omniglot data. A distinguishing feature of AutoBots is that all models are trainable on a single desktop GPU (1080 Ti) in under 48h.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
45 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Traj-MAE: Masked Autoencoders for Trajectory Prediction cites AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
- AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting cites EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning
- AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting cites nuScenes: A Multimodal Dataset for Autonomous Driving
- MotionLM: Multi-Agent Motion Forecasting as Language Modeling cites nuScenes: A Multimodal Dataset for Autonomous Driving
- MotionLM: Multi-Agent Motion Forecasting as Language Modeling cites AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
- ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation cites nuScenes: A Multimodal Dataset for Autonomous Driving
- ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation cites PRANK: motion Prediction based on RANKing
Cited by top-tier papers45
- MotionLM: Multi-Agent Motion Forecasting as Language ModelingAri Seff, Brian Cera, Dian Chen, Mason Ng et al.ICCV 2023 · 186 citations
- Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked AutoencodersJie Cheng, Xiaodong Mei, Ming LiuICCV 2023 · 123 citations
- ADAPT: Efficient Multi-Agent Trajectory Prediction with AdaptationGörkay Aydemir, Adil Kaan Akan, Fatma GüneyICCV 2023 · 85 citations
- Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose EncodingZhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu et al.NeurIPS 2023 · 79 citations
- Traj-MAE: Masked Autoencoders for Trajectory PredictionHao Chen, Jiaze Wang, Kun Shao, Furui Liu et al.ICCV 2023 · 70 citations
Builds on10
- AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent ForecastingYe Yuan, Xinshuo Weng, Yanglan Ou, Kris KitaniICCV 2021 · 658 citations
- EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational ReasoningJiachen Li, Fan Yang, Masayoshi Tomizuka, Chiho ChoiNeurIPS 2020 · 258 citations
- PRANK: motion Prediction based on RANKingYuriy Biktairov, Maxim Stebelev, Irina Rudenko, Oleh Shliazhko et al.NeurIPS 2020 · 33 citations
- TAP: Text-Aware Pre-Training for Text-VQA and Text-CaptionZhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin et al.CVPR 2021
- nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
Related papers
- HiVT: Hierarchical Vector Transformer for Multi-Agent Motion PredictionZikang Zhou, Luyao Ye, Jianping Wang, Kui Wu et al.CVPR 2022 · 379 citations
- SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion ForecastingAlexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl, Horst PosseggerCVPR 2026
- Query-Centric Trajectory PredictionZikang Zhou, Jianping Wang, Yung-Hui Li, Yu-Kai HuangCVPR 2023
- Quantifying Uncertainty in Motion Prediction with Variational Bayesian MixtureJuanwu Lu, Can Cui, Yunsheng Ma, Aniket Bera et al.CVPR 2024 · 6 citations
- SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World ModelJiayuan Du, Yiming Zhao, Zhenglong Guo, Yong Pan et al.CVPR 2026 · 6 citations
